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from the Journal. Dianne has created the cover and interior art and laid out 
the articles and departments in the Journal since its inception in 2014. We will 
miss working with her and wish her the best as she enters her next chapter. 
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ah , Migration 


by Benedict Reuschling I 

Is to be moved around in your ▼ 


Every once in a while, things needs 
system. This is usually to make space for other things, or the new 
location is simply more fitting. This is true with simple files and 
directories, ZFS datasets, and sometimes whole installations. In 
my case, it was a jail that had started its life as a test system and 
I liked the installation so much that I did not want to set it all up 
again when the host it was running on was to be used for some 
other purpose. Luckily, I had set up the jail with iocage as the 
management framework. I knew that it supported migrations 
but had never used it before. Thus, it was a good opportunity to 
put my learning into an article. 


v 


Cold and Hot Migrations 

T here are two kinds of migrations: cold and hot. In a cold migration, 
the system or service is usually shut down for the time of the migra¬ 
tion effort and then started again. A hot migration does not need 
that and can still provide all services and functions it normally provid¬ 
ed. This is usually achieved using a shared medium, and even some clever 
networking tricks, to continue running established network connections. 
There are some variations where a hot migration happens so fast that the 
user does not realize it really was shut down briefly and then started again 
on the new location. Other methods, like a hot-standby system that takes 
over while one of the hosts is migrated, create the illusion of a constantly 
available service. These kinds of failover scenarios require planning and must 
be set up in this way (ideally from the start), which is usually not trivial. 

After the cold migration is done and has been put in its new location, the 
jail can be started again. This process may sound trivial, but there are pit- 
falls. Most of them involve the fact that in the new environment, there will 
be a different network. New interface names and/or IP addresses need to be 
considered after the move. 

In my case, it was a single jail running some services that were not critical 
enough to require the legendary 99.999% uptime. A little bit of downtime 
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was acceptable and the data on the machine was not as big, so downtime would 
be minimal. With iocage, cold migrations are done by creating an archive of the 
ZFS datasets that make up the jail, plus a checksum to verify its integrity. As I 
mentioned earlier, the jail needs to be shut down to have a consistent archive 
without any open sockets or processes still running in the jail's main memory. 

Exporting 

To export a jail in iocage, identify the jail with its UUID by running: 

# iocage list 

H— — —-1—- 1 *4" I * E 

| JID | NAME | STATE | RELEASE I IP4 I 

| 1 | icinga | up I 12.0-RELEASE | 10.0.0.15 | 

4--4--+-4-1-+ 

In this case, it is my icinga monitoring jail, and I can control it using the UUID 
provided in the NAME column. To export that jail, it needs to be shut down first. 
This is done with the iocage stop <uuid> command: 

# iocage stop icinga 

# Stopping icinga 

+ Executing prestop OK 
+ Stopping services OK 
+ Tearing down VNET OK 
+ Removing devfs_ruleset: 5 OK 
+ Removing jail process OK 
+ Executing poststop OK 

Another call to iocage list should confirm that the jail is stopped now: 

+-+-+-+-+-+ 

| JID | NAME | STATE | RELEASE | IP4 | 

| - | icinga | down | 12.0-RELEASE | 10.0.0.15 I 

+-+-+-+-+-+ 

During the installation of iocage, a dataset called images is created among oth¬ 
ers on the pool dedicated for iocage. When executing the migration command, 
the archive making up the jail and the checksum are placed within the images 
directory. 

# iocage export icinga 

Exporting dataset: mypool/iocage/jails/icinga 
Exporting dataset: mypool/iocage/jails/icinga/root 
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Preparing compressed file: /mypool/iocage/images/icinga_2019-10-10.zip. 
Depending on how much data is in the jail, the export process might take 
some time to finish. The same is true for the checksum of the file. As you can 
see from the output, the resulting zip file is named after the jail name plus 
the date of the export. 

Copy the zip file to the new host. Once the copy operation is complete, 
make sure that the checksums still match with the checksum file generated in 
the export step. A simple way to verify the checksum is this: 

# sha256 -r icinga_2019-10-10.zip 

# cat icinga_2019-10-10.zip 

That way, the two checksums are placed one over the other for an easy 
comparison. Next, install iocage on the new host if you have not done so 
already. After activating the pool (using iocage activate), create the directory 
structure using iocage fetch. Once it exists, the zip file must be placed in the 
images directory. From there, iocage can pick it up when running the import 
command: 

# iocage import icinga 

Importing dataset: icinga 
Importing dataset: icinga/root 

Imported: icinga 

The jail will now show up in the iocage list output: 

# iocage list 

+—-1-—-1-1-1-—-F 

| JID | NAME | STATE | RELEASE | IP4 | 

| - | icinga | down | 12.0-RELEASE | 10.0.0.15 | 

+-+-+-+-+-+ 

Depending on your new destination's jail setup, a new IP address must be 
set in order to allow networking for the jail just like before. By default, all the 
settings are preserved when migrating, including the old IP address, which 
may or may not be correct in the new location. Setting a new address might 
be as simple as using "iocage set ip4_addr" to a new IP address. Other 
options iocage supports are jails that share an IP and VNET jails. Check the 
iocage man page for more details. I can also recommend Michael W Lucas's 
book FreeBSD Mastery: Jails, which details a lot of different usage scenarios 
for jails with ready-to-use examples. 

In my case, I was changing the IP address to a whole different network. 
Another run of iocage list will confirm that the settings were applied: 
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# iocage list 

H-1-1-1-(-f- 

| JID | NAME | STATE | RELEASE | IP4 | 

I - | icinga | down | 12.0-RELEASE | 192.168.1.128 | 

+-+-+-+-+-+ 

Starting the Jail 

We can then start the jail using iocage start. Make sure that the jail in the old 
location is still in the down state, or both of them will vie for control of their IP 
address. Once the jail has started, make sure that the services running in it are 
listening to the new address. This usually includes changing the respective con¬ 
figuration files and then restarting the service to apply them. Once you are satis¬ 
fied with the result and the jail is running in the new location, the old jail can be 
archived (backed up) or simply deleted. This frees up some space on the old host 
location. Don't forget the exported files in the iocage/images directory. 

Jail migrations are not as scary as they may sound. With a bit of preparation 
and time to transfer the resulting files, it is a viable solution without having to 
set up jails completely from scratch again. • 


T Biiedict Reuschling joined the FreeBSD Project in 2009. After receiving his full documentation 
commit bit in 2010, he actively began mentoring other people to become FreeBSD committers. Fie joined 
the FreeBSD Foundation in 2015, where he is currently serving as vice president. Benedict has a Master 
of Science degree in Computer Science and is teaching a UNIX for software developers class at the 
Darmstadt University of Applied Sciences, Darmstadt, Germany. Together with Allan Jude, he is host of 
the weekly BSDNow. tv (httpU/BSPNovy tv) podcast. 
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Virtual 


Modern FreeBSD offers a range of 
virtualization options, from the 
traditional jail environment shar¬ 
ing the network stack with the 
host operating system, over vnet 
jails, which allow each jail to have 
its own network stack, to bhyve 
virtual machines running their 
own kernels/operating systems. 
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by Michael Gmelin 


D epending on individual requirements, there are different ways to configure 
the virtual network. Jail and VM management tools can ease the process by 
abstracting away (at least some of) the underlying complexities. 


Document Conventions 

This article is based on FreeBSD 12.1-RELEASE, the latest release version of 
FreeBSD at the time of writing. It assumes that ZFS is used on hosts running jails 
and bhyve VMs. 

Unless noted otherwise, packages used in examples are from FreeBSD's quarterly 
branch 2019Q4 ( https://svnweb.freebsd.org/ports/branches/2019Q4/) . Using the 
quarterly package repository is the default configuration on a freshly installed 
FreeBSD system. 

All examples are limited to IPv4 for the sake of simplicity. 

This article introduces various jail and VM management tools, even though 
everything shown could be configured directly without installing any packages by 
modifying system configuration files like /etc/jail.conf manually. 

Code and terminal interaction/output is formatted like this. If you would 
like to copy and paste code you see in this article, you may do so at 
https://bloq.qrem.de/ayvn . 

Sometimes (but not always) the output of commands is shown in examples for 
clarity. 

Man page references are formatted in italics, specifying the manual section in 
parentheses; e.g., security(7) refers to the security man page, which can be dis¬ 
played by typing man 7 security, or man security, as there are no other 
man pages of that name in the manual. 


: p: 


FreeBSD Journal 

• • 




































































FreeBSD package references are formatted in italics, using the port's origin, 
which consists of <category>/<name>, e.g., security/sudo, which can be 
installed as a binary package (pkg install sudo) or from ports 
(cd /usr/ports/security/sudo && make install clean). 

License 

This work is licensed under a Creative Commons Attribution 4.0 International 
License (CC BY 4.0) (https://creativecommons.Org/licenses/by/4. 0/). 

Plain Jails 

Plain (as in non-VNET) jails share the network stack with the jailhost they're run¬ 
ning on. Therefore, network configuration and firewalling are done on the jail- 
host and not inside of the jail. They are the traditional way of creating jails and 
despite their limitations still very useful when it comes to containing software, 
filesystems, and services. E.g., ports-mgmt/poudriere, FreeBSD's bulk package 
builder and port tester, makes heavy use of plain jails. 

Plain Jails Using Inherited IP Configuration 

The most basic option is running a jail that inherits its network configuration (all 
interfaces/IP addresses) from the jailhost. This is a common setup if the jail is 
mostly used as a container to keep the jailhost (the host the jails run on) "clean" 
of dependencies. This way the host only requires a minimal number of packages 
(basics like security/sudo, shells/bash, and sysutils/tmux ), while application jails 
can be snapshotted, backuped, and managed/moved separately. 

Example of how to configure, run, and destroy a simple jail using 
sysutils/iocage: 


root®jailhost:~ # pkg install py36-iocage 


root®jailhost:~ 
ZFS pool 'zroot 
root®jailhost:~ 


# iocage activate zroot 
successfully activated. 

# iocage create -r 12.1-RELEASE -n simplecage ip4=inherit 


simplecage successfully created! 

root®jailhost:~ # iocage console -f simplecage 

root®simplecage:~ # fetch -q -o - http://canhazip.com 
<your public facing IP shown here> 
root®simplecage:~ # logout 

root®jailhost:~ # iocage destroy -f simplecage 


Stopping simplecage 
Destroying simplecage 
root®jailhost:~ # 

Note: This inherits all interfaces and copies the resolver configuration from the 
jailhost. 
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Same example using sysutils/pot, an alternative jail manager that aims to provide 
container-like features and sysutils/nomad integration: 

root@jailhost:~ # pkg install pot 
root@jailh.ost: ~ # pot init 
root@jailhost:~ # pot create-base -r 12.1 
root@jailhost:~ # pot create -p simplepot -b 12.1 
root@jailhost:~ # pot run simplepot 

root@simplepot:~ # fetch -q -o - http://canhazip.com 
<your public facing IP shown here> 
root@simplepot:~ # exit 

root@jailhost:~ # pot destroy -Fp simplepot 
===> Destroying pot simplepot 
root®jailhost:~ # 

Note: sysutils/pot is an evolving project, so you might want to grab the version 
from FreeBSD's latest branch by modifying /etc/pkg/FreeBSD.conf or install it from 
ports (/usr/ports/sysutils/pot). Make sure to set POT_EXTIF in 
/ usr/local/etdpot/pot.cfg in case your network interface name isn't "emO". 


Plain Jails Using a Dedicated IP Address 

In case a bit more separation is wanted, a dedicated static IP address can be 
assigned. This prevents a jail from listening to ports used by the jailhost, e.g., per¬ 
mitting to ssh(l) directly into a jail listening to the standard port (22). 

In the example below, the jailhost uses 192.168.0.2 as its primary IP address, 
192.168.0.1 as the default gateway, and 192.168.0.3 is added as an additional IP 
address to be used by the jail. The goal is to create a jail on its dedicated static IP 
address on the local network and run sshd(8) inside. 

This assumes that the jailhost has its secure shell daemon configured to listen 
only to the relevant IP address, by setting ListenAddress to 192.168.0.2 in 
/etc/ssh/sshd_config and reloading it by running service sshd reload. 

Static LAN IP jail example using sysutils/pot. 

root@jailhost:~ # pot create -p aliaspot -b 12.1 -N alias -i 192.168.0.3 

===> Creating a new pot 

===> pot name : aliaspot 

===> type : multi 

===> base : 12.1 

===> pot_base : 

===> level : 1 

===> network-type: alias 

===> ip : 192.168.0.3 

===> dns : inherit 

root@jailhost:~ # pot run aliaspot 
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Continued 


root©aliaspot:~ # service sshd enable 
sshd enabled in /etc/rc.conf 
rootOaliaspot:~ # service sshd start 


rootOaliaspot:~ # exit 


root©jailhost:~ # 

sockstat 

; -41j aliaspot 


USER 

COMMAND 

PID 

FD PROTO 

LOCAL ADDRESS 

FOREIGN 

root 

sshd 

7324 

3 tcp4 

192.168.0.3:22 

*: * 

root 

sendmail 

7183 

3 tcp4 

192.168.0.3:25 

*: * 

root 

syslogd 

7102 

5 udp4 

192.168.0.3:514 

*: * 

root©jailhost:~ # 

pot destroy -Fp 

aliaspot 



===> Destroying pot aliaspot 
rootOjailhost:~ # 


You can run ifconfig(8) in various stages of the provisioning process to under¬ 
stand when the IP alias (192.168.0.33/32) is added/removed to/from the interface. 
Depending on the use case, it might also make sense to add aliases permanently 
in the jailhost's /etc/rc.conf. 

Note: In such a static single IP configuration, the jail's only IP address also (magi¬ 
cally) serves as localhost, which can be confusing at times and also means that 
services that usually listen to localhost and therefore are not reachable from the 
outside world (such as sendmail_submit in the example above) are suddenly 
exposed. So, it's important to firewall services on the jailhost properly in such a 
setup (following the best practice of blocking all traffic by default). It's possible to 
add unique loopback addresses (like 127.0.0.2/8) to a jail, but given the additional 
complexity this introduces, doing so is only worth the effort in a limited number 
of use cases. 


Plain Jails Using a VLAN IP Address 

A variation of the previous setup is to configure the jail to listen to IP addresses on 
a dedicated (private) network, either by using a dedicated interface or by adding 
VLAN interfaces to an existing physical interface. This provides a better segmenta¬ 
tion of the network and allows to deploy central filtering, outbound network 
address translation (NAT), and inbound redirection (DNAT) of network traffic. 

In the example below, a VLAN interface (VLAN tag 101, 10.1.1.1/24) is config¬ 
ured on the jailhost (interface "emO") and put to use in a jail configured using 
sysutils/iocage: 


root©jailhost:~ # sysrc ifconfig_em0_101="10.1.1.1/24" 
ifconfig_em0_101: -> 10.1.1.1/24 

root©jailhost:~ # ifconfig emO.101 create 
root©jailhost:~ # iocage create \ 

-r 12.1-RELEASE -n vlancage ip4_addr="em0.101110.1.1.2" 

vlancage successfully created! 

root©jailhost:~ # iocage console -f vlancage 


Continues next page 


Nov/Dec 2019 


11 






















Continued 

root@vlancage:~ # ifconfig -g vlan 
emO.101 

root@vlancage:~ # ifconfig emO.101 

emO.101: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST metric 0 \ 
mtu 1500 

options=3<RXCSUM,TXCSUM> 
ether f4:4d:30:aa:bb:cc 

inet 10.1.1.2 netmask Oxffffffff broadcast 10.1.1.2 
groups: vlan 

vlan: 101 vlanpcp: 0 parent interface: emO 
media: Ethernet autoselect (lOObaseTX <full-duplex>) 
status: active 
root@vlancage:~ # logout 

root®jailhost:~ # iocage destroy -f vlancage 

Stopping vlancage 
Destroying vlancage 
root®jailhost:~ # 

This assumes that firewalling and NAT happen on a different host/appliance on 
your network. As the host already has an IP address configured (10.1.1.1/24), 
adding the jail's IP address as an alias host address (netmask /32) is exactly what 
was wanted. This also helps configuration-wise, as it allows things like configuring 
static routes before jails are started and setting up firewall rules based on the inter¬ 
face's network configuration. 

In case the jail should use the interface's primary IP address, it's important to con¬ 
figure the IP address and netmask exactly like they're configured on the VLAN inter¬ 
face. In the example above this would mean setting 

ip4_addr="em0.101110.1.1.1/24", iocage(8) is smart enough to not remove 
the jail's IP address from an interface if it was already configured prior to starting 
the jail. 

Hint: Like most network interfaces, it's possible to name VLAN interfaces semanti¬ 
cally; see rc.conf(5) (man rc.conf) for details. 

Plain Jails Using a Loopback IP Address 

Under some circumstances, e.g., if the number of IP addresses you can assign to 
your LAN interface is limited and/or all you are maintaining is a single jailhost, it 
might make sense to have the jail listen to a local loopback interface. In this case 
you usually use a local firewall to NAT outbound traffic and redirect inbound traffic 
(if necessary). 

In the sysutils/iocage example below, a dedicated local interface named lol , hold¬ 
ing the IP address 172.31.255.17/32, is created to serve the jail. In this example the 
IP address is already assigned by the jailhost on boot and not by sysutils/iocage 
when starting the jail: 


root®jailhost:~ # sysrc cloned_interfaces+="lol" 
cloned_interfaces: -> lol 

root@jailhost:~ # sysrc ifconfig_lol="172.31.255.17/32" 
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Continued 


ifconfig_lol: -> 172.31.255.17/32 

root@jailhost:~ # ifconfig lol create 
root@jailhost:~ # ifconfig lol 

lol: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST metric 0 mtu 16384 

options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> 
inet 172.31.255.17 netmask Oxffffffff 
groups: lo 

nd6 options=29<PERF0RMNUD,IFDISABLED,AUT0_LINKL0CAL> 
rootOjailhost:~ # iocage create \ 

-r 12.1-RELEASE -n Iocage ip4_addr="lol|172.31.255.17/32" 

Iocage successfully created! 
root@jailhost:~ # iocage console -f Iocage 

root@locage:~ # host freebsd.org 

;; connection timed out; no servers could be reached 
rootOlocage:~ # logout 
root@jailhost:~ # 

Adding Outbound NAT for Public Traffic 

At this point the jail exists but has no way to communicate with the outside 
world. This can be fixed by adding an outbound NAT rule to the firewall. 

Warning: Enabling/configuring a firewall is a great way to lock yourself out of a 
machine. Make sure you have a plan in place on how to regain access in case this 
happens to you accidentally 

In this example the pf(4) (packet filter) firewall is used. It is assumed that there 
is no firewall configured yet. 

The following /etc/pf.conf is created: 


ext_if="emO" 

www_jail="172.31.255.17" 

# outbound NAT for www jail on jailhost's primary IP address 
nat on $ext_if from $www_jail -> $ext_if:0 

# redirect www traffic into jail 

rdr on $ext_if proto tcp to $ext_if:0 port www -> $www_jail 

# don't interfere with jailhost's localloop 
set skip on loO 

# prevent spoofing 
antispoof for loO 
antispoof for lol 
antispoof for $ext_if 
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Continued # block all by default (best practice) 
block 

# allow any traffic from jail that doesn't go to the jailhost 

# (includes traffic inside of the jail to itself) 
pass from $www_jail to !$ext_if:0 

# allow access to web server 

pass proto tcp to $www_jail port www 

# allow access to manage host 

pass in on $ext_if proto tcp to $ext_if:0 port ssh 

# allow outbound traffic 
pass out on $ext_if 


Next, pf(4) is enabled and outbound connectivity is verified to work as expected: 


root©jailhost:~ # service pf enable 
pf enabled in /etc/rc.conf 
root©jailhost:~ # service pf start 
... (session drops) ... 

root©jailhost:~ # iocage console -f locage 
root©locage:~ # fetch -q -o - http://canhazip.com 
<your public facing IP shown here> 
root@locage:~ # logout 
root©j ailhost:~ # 

Running a Service and Redirecting Traffic to It 

Now that there is outbound connectivity within the jail, let's install www/nginx 
inside to serve static content. The necessary firewall rules are already in place 
(check the rdr and pass rules that were configured earlier): 


root©jailhost:~ # grep -B1 "port www" /etc/pf.conf 
# redirect www traffic into jail 

rdr on $ext_if proto tcp to $ext_if:0 port www -> $www_jail 


# allow access to web server 
pass proto tcp to $www_jail port www 
root@jailhost:~ # iocage console -f locage 
root@locage:~ # pkg install nginx 
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root©locage:~ # rm /usr/local/www/nginx 
root@locage:~ # mkdir /usr/local/www/nginx 

root@locage:~ # echo "Hello Jail" >/usr/local/www/nginx/index.html 

root©locage:~ # service nginx enable 

nginx enabled in /etc/rc.conf 

root@locage:~ # service nginx start 

Performing sanity check on nginx configuration: 

nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok 
FreeBSD Journal Continues next page 


















Continued 


nginx: configuration file /usr/local/etc/nginx/nginx.conf test is\ 

successful 

Starting nginx. 

rootOlocage:~ # logout 

root@jailh.ost: ~ # fetch -q -o - http://172.31.255.17 

Hello Jail 

root®j ailhost:~ # 

Accessing the web server hosted within the jail from the outside should work 
at this point and can be verified by pointing a web browser to the server's primary 
IP address. 

Warning: Even though it's possible to create clean and easy to understand 
configurations this way, these setups don't scale very well and can get quite a 
hassle to maintain. Also, even though it's possible to firewall between the various 
jails to a certain extent, it's not very practical and therefore VNET jails should be 
preferred. 

VNET Jails and bhyve VMs 

VNET(9) —the network subsystem virtualization infrastructure—is a technology to 
virtualize the network stack. 

Even though VNET first appeared in FreeBSD 8.0, it was considered an experi¬ 
mental feature until recently. With the arrival of FreeBSD 12 it is finally available 
to everyone without the burden of building a custom kernel. 

When applied to jails, VNET allows each jail to have its own network stack. This 
solves a couple of the shortcomings we've seen so far and gets us improvements 
like better segregation and isolation, a "normal" local loopback interface, and the 
possibility to run an independent firewall inside a jail. 

VNET Jails Using sysutils/pot 

This section will demonstrate how to create multiple VNET jails using sysutils/pot. 
Before demonstrating how this is done, the configuration changes to pf(4) done 
above are reverted (in case you didn't run these examples and pf(4) isn't running, 
please replace service pf reload with service pf start): 


root@jailhost:~ # rm -f /etc/pf.conf 
root@jailhost:~ # pot init 

Please, check that your PF configuration file is still valid! 

root@jailhost:~ # cat /etc/pf.conf 

nat-anchor pot-nat 

rdr-anchor "pot-rdr/*" 

root@jailhost:~ # service pf enable 

root@jailhost:~ # service pf reload 


pot automatically uses VNET when a jail is created with a network-type of 
public-bridge. Based on the "Internal Virtual Network configuration" in 
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/usr/local/etc/pot/pot.cfg it creates a bridge interface and assigns an IP address that 
serves as the default gateway for jails to it. 


root®jailhost:~ # pot vnet-start 

pfctl: pf already enabled 

root®jailhost:~ # ifconfig bridgeO 

bridgeO: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST metric 0\ 
mtu 1500 

ether 02:0e:35:b7:6d:00 

inet 10.192.0.1 netmask OxffcOOOOO broadcast 10.255.255.255 
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 
groups: bridge 
nd6 options=l<PERFORMNUD> 


On jail creation, pot automatically assigns a static IP address from the configured 
range. On jail start, an epair(4) interface is created and its "a-side" is added to the 
bridge: 


root®jailhost:~ # pot 
root®jailhost:~ # pot 
root®jailhost:~ # pot 
root®jailhost:~ # pot 


create -b 12.1 -N public-bridge -p vnetpotl 
create -b 12.1 -N public-bridge -p vnetpot2 
start vnetpotl 
start vnetpot2 


root®jailhost:~ # pot list 
pot name : base-12_l 

network : inherit 
active : false 


pot name : vnetpotl 

network : public-bridge 
ip : 10.192.0.3 
active : true 


pot name : vnetpot2 

network : public-bridge 
ip : 10.192.0.4 
active : true 

root®jailhost:~ # ifconfig bridgeO 

bridgeO: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0\ 
mtu 1500 

ether 02:Oe:35:b7:6d:00 

inet 10.192.0.1 netmask OxffcOOOOO broadcast 10.255.255.255 
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 
member: epairla flags=143<LEARNING,DISCOVER,AUTOEDGE,AUT0PTP> 
ifmaxaddr 0 port 7 priority 128 path cost 2000 
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member: epairOa flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> 
ifmaxaddr 0 port 6 priority 128 path cost 2000 
groups: bridge 
nd6 options=l<PERFORMNUD> 
root@jailhost:~ # ifconfig -g epair 
epairOa 
epairla 

root@jailhost:~ # ifconfig epairOa 

epairOa: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST>\ 
metric 0 mtu 1500 

options=8<VLAN_MTU> 
ether 02:f5:dd:bc:cd:0a 
groups: epair 

media: Ethernet lOGbase-T (lOGbase-T <full-duplex>) 
status: active 

nd6 options=29<PERF0RMNUD,IFDISABLED,AUT0_LINKL0CAL> 
root@jailhost:~ # ifconfig epairla 

epairla: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST>\ 
metric 0 mtu 1500 

options=8<VLAN_MTU> 
ether 02:d8:52:a6:86:0a 
groups: epair 

media: Ethernet lOGbase-T (lOGbase-T <full-duplex>) 
status: active 

nd6 options=29<PERF0RMNUD,IFDISABLED,AUT0_LINKL0CAL> 


The epair(4) interface's "b-side" is used by the jail, which sets the configured 
static IP address as configured in its /etc/rc.conf. 


root@jailhost:~ # jexec vnetpotl grep ifconfig /etc/rc.conf 
ifconfig_epairOb="inet 10.192.0.3 netmask 255.192.0.0 
root@jailhost:~ # jexec vnetpotl ifconfig epairOb 

epairOb: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST metric 0\ 
mtu 1500 

options=8<VLAN_MTU> 
ether 02:f5:dd:bc:cd:0b 

inet 10.192.0.3 netmask OxffcOOOOO broadcast 10.255.255.255 
groups: epair 

media: Ethernet lOGbase-T (lOGbase-T <full-duplex>) 
status: active 

nd6 options=29<PERF0RMNUD,IFDISABLED,AUT0_LINKL0CAL> 
root@jailhost:~ # jexec vnetpot2 grep ifconfig /etc/rc.conf 
ifconfig_epairlb="inet 10.192.0.4 netmask 255.192.0.0" 
root@jailhost:~ # jexec vnetpot2 ifconfig epairlb 

epairlb: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST metric 0\ 
mtu 1500 

options=8<VLAN_MTU> 
ether 02:d8:52:a6:86:0b 

inet 10.192.0.4 netmask OxffcOOOOO broadcast 10.255.255.255 
groups: epair 

media: Ethernet lOGbase-T (lOGbase-T <full-duplex>) 
status: active 

nd6 options=29<PERF0RMNUD,IFDISABLED,AUT0_LINKL0CAL> 
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Using the anchors configured in /etc/pf.conf, pot adds NAT rules to pf(4), which 
can be inspected using pfctl(8'): 


root@jailhost:~ # pfctl -s nat -a pot-nat 

nat on emO inet from 10.192.0.0/10 to any -> (emO) round-robin 


Like in the plain jail redirect example above, we want one of the VNET jails to 
run a web server and make it available to the outside world through an inbound 
redirect (DNAT). Let's install www/nginx inside of the first jail and use pot's 
"export-port" feature to create the inbound redirect: 

root@jailhost:~ # pot run vnetpotl 
root©vnetpotl:~ # pkg install nginx 

rootOvnetpotl:~ # rm /usr/local/www/nginx 
root@vnetpotl:~ # mkdir /usr/local/www/nginx 

root@vnetpotl:~ # echo "Hello Jail" >/usr/local/www/nginx/index.html 
rootOvnetpotl:~ # service nginx enable 
nginx enabled in /etc/rc.conf 
rootOvnetpotl:~ # service nginx start 

root©vnetpotl:~ # exit 

root@jailh.ost: ~ # fetch -q -o - http://10.192.0.3 
Hello Jail 

root@jailhost:~ # pot export-ports -p vnetpotl -e 80:80 
root@jailhost:~ # pot stop vnetpotl 
root@jailhost:~ # pot start vnetpotl 

root@jailhost:~ # pot show 
pot vnetpotl 

disk usage : 55.2M 

===> runtime memory usage require rctl enabled 

Network port redirection 

192.168.0.2 port 80 -> 10.192.0.3 port 80 

pot vnetpot2 

disk usage : 308K 

===> runtime memory usage require rctl enabled 
root©jailhost:~ # 


pot creates an anchor containing redirect pass rules for exported ports in pf(4), 
which again can be inspected using pfctl(8,)\ 


root@jailhost:~ # pfctl -a pot-rdr -s Anchors 
pot-rdr/vnetpot1 

root@jailhost:~ # pfctl -a pot-rdr/vnetpotl -s nat 
rdr pass on emO inet proto tcp \ 

from any to 192.168.0.2 port = http -> 10.192.0.3 port 80 
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Note: At this point pf(4) is only used for NATting and redirecting traffic; it does 
not block any traffic. 


VNET Jails Using sysutils/iocage 

Compared to sysutils/pot, sysutils/iocage offers more flexibility and fine-grained 
control when creating custom setups. 

Managing Bridges 

iocage(8) expects the user to manage the bridges VNET jails connect to, so usually 
this is done manually, using ifconfig(8) and modifying /etc/rc.conf. 

An alternative way to manage bridges is to use sysutils/vm-bhyve, which 
abstracts them as "virtual switches". As vm-bhyve(8) is a great tool to manage 
bhyve VMs, we'll take advantage of it to manage bridges for iocage(8) jails and 
later connect bhyve VMs to them. 


rootOjailhost:~ # pkg install vm-bhyve 
vm enabled in /etc/rc.conf 
rootOjailhost:~ # service vm enable 
vm enabled in /etc/rc.conf 

rootOjailhost:~ # sysrc vm_dir=zfs:zroot/vms 
vm_dir: -> zfs:zroot/vms 

rootOjailhost:~ # zfs create zroot/vms 
rootOjailhost:~ # vm init 
rootOjailhost:~ # vm help | grep switch 


Note: vm-bhyve(8) supports different switch types. We limit ourselves to the 
default "standard" for the time being. 


Running reorder /usr/local/etc/rc .d/* shows that vm is executed 
prior to iocage on system startup. This means that bridges will be available when 
jails are connected to them. 

With that sorted out, we create a first switch: 


rootOjailhost:~ # vm switch create -a 10.1.1.1/24 services 
rootOjailhost:~ # vm switch list 

NAME TYPE IFACE ADDRESS PRIVATE MTU VLAN PORTS 

services standard vm-services 10.1.1.1/24 no - 

rootOjailhost:~ # ifeonfig vm-services 

vm-services: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0\ 
mtu 1500 

ether f6:b5:a8:80:78:5e 

inet 10.1.1.1 netmask OxffffffOO broadcast 10.1.1.255 
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 
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root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 
groups: bridge vm-switch viid-10cd3@ 
nd6 options=l<PERF0RMNUD> 


and connect a new iocage(8) jail to it: 


root@jailh.ost: ~ # iocage create -r 12.1-RELEASE -n vnetcage \ 
interfaces="vnetO:vm-services" ip4_addr="vnet0110.1.1.2/24" \ 
defaultrouter="10.1.1.1" vnet_default_interface="vm-services" \ 
vnet=on 

vnetcage successfully created! 
root@jailh.ost: ~ # iocage start vnetcage 
No default gateway found for ipv6. 

* Starting vnetcage 
+ Started OK 

+ Using devfs_ruleset: 5 
+ Configuring VNET OK 
+ Using IP options: vnet 
+ Starting services OK 
+ Executing poststart OK 
root@jailhost:~ # ifconfig -g epair 
vnetO.29 

root@jailhost:~ # ifconfig vnetO.29 

vnetO.29: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST>\ 
metric 0 mtu 1500 

description: associated with jail: vnetcage as nic: epairOb 

options=8<VLAN_MTU> 

ether f4:4d:30:13:le:5c 

hwaddr 02:82:25:aO:cf:Oa 

inet6 fe80: :f64d:30ff :fel3: le5c°/ 0 vnetO. 29 prefixlen 64 scopeid 0x4 
groups: epair 

media: Ethernet lOGbase-T (lOGbase-T <full-duplex>) 
status: active 

nd6 options=21<PERF0RMNUD,AUT0_LINKL0CAL> 
root@jailhost:~ # ifconfig vm-services 

vm-services: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST metric 0\ 
mtu 1500 

ether 0a:3b:ea:3e:5c:eb 

inet 10.1.1.1 netmask OxffffffOO broadcast 10.1.1.255 
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 
member: vnetO.29 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUT0PTP> 
ifmaxaddr 0 port 4 priority 128 path cost 2000 
groups: bridge vm-switch viid-10cd3@ 
nd6 options=l<PERFORMNUD> 
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root®jailhost:~ # iocage console vnetcage 
rootSvnetcage:~ # fetch -q -o - http://canhazip.com 
fetch: http://canhazip.com: No address record 
rootSvnetcage:~ # host canhazip.com 

;; connection timed out; no servers could be reached 
rootSvnetcage:~ # logout 
root®jailhost:~ # 

To allow that jail to talk to the world, we'll add minimal configuration to add 
outbound NAT for it (this assumes pf(4) isn't running already): 


rootSjailhost:~ # echo "set skip on loO" >/etc/pf.conf 

rootSjailhost:~ # echo "nat on emO from 10.1.1/24 -> em0:0" >>/etc/pf.conf 

root@jailhost:~ # service pf enable 

pf enabled in /etc/rc.conf 

rootSjailhost:~ # service pf start 

Enabling pf. 

root@jailhost:~ # iocage console vnetcage 
rootSvnetcage:~ # fetch -q -o - http://canhazip.com 
<your public facing IP shown here> 
rootSvnetcage:~ # logout 
root®j ailhost:~ # 


VM running FreeBSD is 
added to the same virtual 
switch, using a different IP 
address on network 
10.1.1.0/24. 

Network diagram: 
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To ease the provisioning process, we make the bhyve VM get its IP address over 
DHCP, dns/dnsmasq is used for this purpose: 


root@jailhost:~ # pkg install dnsmasq 

cat >/usr/local/etc/dnsmasq.conf <<E0F 

domain-needed 

listen-address=10.1.1.1 

interface=vm-services 

bind-interfaces 

local-service 

dhcp-authoritative 

dhcp-range=10.1.1.10,10.1.1.20 

EOF 

root@jailh.ost: ~ # service dnsmasq enable 
dnsmasq enabled in /etc/rc.conf 
root@jailh.ost: ~ # service dnsmasq start 


We already installed sysutils/vm-bhyve earlier; all that's left to do is download the 
install ISO, create the VM, connect it to the switch, and run the installer: 


root@jailhost:~ # vm iso https://download.freebsd.org/ftp/releases/\ 
ISO-IMAGES/12.I/FreeBSD-12.l-RELEASE-amd64-bootonly.iso 

root@jailhost:~ # vm create guest 

root@jailhost:~ # vm add -d network -s services guest 
root@jailhost:~ # vm install guest \ 

FreeBSD-12.l-RELEASE-amd64-bootonly.iso 
root@jailhost:~ # vm console guest 


root@jailhost:~ # vm stop guest 
root@jailhost:~ # vm start guest 

This uses the bootonly ISO (network based installation), which should work ok, as 
the IP address is assigned to the guest by dnsmasq(8) (which also provides DNS 
service) and NAT is already configured on the host firewall. 

Note: This shows two network interfaces, vtnetl is the one to use in our setup. 
vtnetO can be removed manually by modifying the VM's configuration file 
/zroot/vms/guest/guest. conf. 


As expected, dnsmasq(8) assigned an IP address from the DHCP pool to the new 
VM (10.1.1.11/24 in this example). In an environment where having a stable IP is 
important, it's advantageous to assign fixed IP addresses that aren't part of the 
dynamic pool based on the virtual network interface's MAC address in 
/usr/local/etc/dnsmasq. conf, e. g., 


dhcp-host=l1:22:33:44:55:66,192.168.0.60 
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Once we added a non-privileged user and enabled sshd(8) in the new VM, we 
are able to ssh(1) into it: 


root@jailhost:~ # ssh user@10.1.1.11 

Are you sure you want to continue connecting (yes/no)? yes 

Warning: Permanently added '10.1.1.11' (ECDSA) to the list of known hosts. 

Password for userOguest: 

$ fetch -q -o - http://canhazip.com 
<your public facing IP shown here> 

$ ~DConnection to 10.1.1.11 closed. 
rootOjailhost:~ # 

Note: By exposing /dev/bpf to a VNETjail, it's possible to configure the jail's IP 
address using DHCP just like it's done for VMs here. Using iocage(8) this is easily 
accomplished by enabling the dhcp property. 

Preventing Traffic Between VNET JailsA/Ms 

A simple way to prevent traffic between jails and VMs is to set the private flag on 
bridge members (ifconfig <bridgename> private <interfacename>). 

Any interface marked as private won't be able to talk to any other interface that 
is also marked as such. 

sysutils/vm-bhyve sets the private flag automatically on vra start if the switch 
was configured to be private beforehand by running 
vm switch private <switchname> on. 

Unfortunately, this is only true for VMs, so if you connect iocage jails to the 
switch, you'll have to set the private flag for them manually on every jail start. 

In the current setup, "vnetcage" can ssh into "guest": 

root@jailhost:~ # iocage console -f vnetcage 
root@vnetcage:~ # nc 10.1.1.11 22 
SSH-2.0-0penSSH_7.8 FreeBSD-20180909 
~C 

root@vnetcage:~ # logout 
root®j ailhost:~ # 


After changing the switch to private mode, the VM's tap interface connected to 
the bridge is marked as PRIVATE. As the jail's epair interface is not tagged as PRI¬ 
VATE, traffic can still flow and ssh still works: 

root@jailhost:~ # vm stop guest 
Sending ACPI shutdown to guest 

rootOjailhost:~ # vm switch private services on 
root@jailhost:~ # vm switch list 

NAME TYPE IFACE ADDRESS PRIVATE MTU VLAN PORTS 

services standard vm-services 10.1.1.1/24 yes - - 

rootOjailhost:~ # vm start guest 
Starting guest 

* found guest in /zroot/vms/guest 


Continues next page 


Nov/Dec 2019 


23 










































Continued 


* booting... 

root@jailhost:~ # ifconfig vm-services 

vm-services: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST^ metric 0\ 
mtu 1500 

ether 0a:3b:ea:3e:5c:eb 

inet 10.1.1.1 netmask OxffffffOO broadcast 10.1.1.255 
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 
member: tapl flags=943<LEARNING,DISCOVER,PRIVATE,AUTOEDGE,AUT0PTP> 
ifmaxaddr 0 port 6 priority 128 path cost 2000000 
member: vnet0.31 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUT0PTP> 
ifmaxaddr 0 port 7 priority 128 path cost 2000 
groups: bridge vm-switch viid-10cd3@ 
nd6 options=l<PERF0RMNUD> 
root@jailhost:~ # iocage console -f vnetcage 
root@vnetcage:~ # nc 10.1.1.11 22 
SSH-2.0-0penSSH_7.8 FreeBSD-20180909 
~C 

root@vnetcage:~ # logout 
root©j ailhost:~ # 


Let's change the jail's connection manually to "private" and perceive that con¬ 
nection attempts to the jail time out, while NAT to the outside world is still intact 
and the jailhost can still connect to the VM via ssh(l)\ 


root@jailhost:~ # ifconfig vm-services private vnet0.31 
root@jailhost:~ # iocage console -f vnetcage 
root@vnetcage:~ # nc -vw 10 10.1.1.11 22 

nc: connect to 10.1.1.11 port 22 (tcp) failed: Operation timed out 

root@vnetcage:~ # fetch -q -o - http://canhazip.com 

<your public facing IP shown here> 

root@vnetcage:~ # logout 

root@jailhost:~ # nc 10.1.1.11 22 

SSH-2.0-0penSSH_7.8 FreeBSD-20180909 

~C 

root@jailhost:~ # 


Ideally, iocage(8) would support a configuration option to allow setting the pri¬ 
vate flag on the bridge member automatically. Until such a feature becomes avail¬ 
able, the script below can be configured to be executed by the jail's poststart hook 
to accomplish (almost) the same. 

/usr/local/sbin/setJoc_vnet_priva te. sh : 
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#!/bin/sh 


set -e 

if [ "$#" -ne 1 -a "$#" -ne 2 ]; then 

echo "Usage: $0 jailname [vnetprefix]" >&2 
exit 1 
fi 

JAILNAME=$1 

VNETPREFIX=${2:-vnetO} 

JAILINFO=$(jls -j ioc-$JAILNAME) 

JID=$(echo "$JAILINFO" | grep $JAILNAME | awk '{ print $1 >') 

for BRIDGE in $(ifconfig -g bridge); do 
C0NFIG=$(if config $BRIDGE) 
set +e 

echo "$C0NFIG" I grep "member: ${VNETPREFIX}\."$JID >/dev/null 

N0TF0UND=$? 

set -e 

if [ $N0TF0UND -eq 0 ]; then 

ifconfig $BRIDGE private $VNETPREFIX.$JID 
exit 0 
fi 

done 

echo "Couldn't find interface $VNETPREFIX.$JID on any bridges" >&2 
exit 1 


The script takes the jail's name as a mandatory parameter and optionally the 
vnet interface (in iocage's internal enumeration). The latter defaults to "vnetO". 

Now, let's restart the "vnetcage" jail, check the bridge member configuration, 
then alter its configuration to make use of the new script, restart again, and com¬ 
pare the resulting bridge configuration: 


root@jailhost:~ # iocage restart vnetcage 

rootOjailhost:~ # ifconfig vm-services | grep vnet 

member: vnetO.15 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUT0PTP> 
rootOjailhost:~ # ifconfig vm-services | grep vnet 
root@jailhost:~ # iocage set \ 

exec_poststart="/usr/local/sbin/set_ioc_vnet_private.sh vnetcage" \ 
vnetcage 

exec_poststart: /usr/bin/true -> /usr/local/sbin/set_ioc_vnet_private.sh \ 
vnetcage 

rootOjailhost:~ # iocage restart vnetcage 
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Continued 

rootOjailhost:~ # ifconfig vm-services | grep vnet 

member: vnet0.16 flags=943<LEARNING,DISCOVER,PRIVATE,AUTOEDGE,\ 

AUT0PTP> 

rootOjailhost:~ # 

As you can see, the bridge member is now correctly configured to be private on 
jail start. 

Note: This solution isn't perfect, as traffic is possible in the short period of time 
between starting the jail and running the execjaoststart command. 

Firewalling Inside VNET JailsA/Ms 

Firewalling inside bhyve VMs is straight-forward—simply run the host firewall of the 
OS running inside the VM. 

Running a firewall inside a jail is a bit more complicated. ipfw(8) is the recom¬ 
mended option, but pf(4) should work at this point as well. For the sake of not 
introducing even more syntax in this article, we'll describe the latter here, even 
though iocage's documentation recommends otherwise. 

To run pf(4) inside a VNET jail, the pf(4) kernel module has to be loaded and vari¬ 
ous devices need to be exposed. This is done by adding a new ruleset set to 
/etc/devfs.rules: 


[vnet_jail_pf=501] 

add include $devfsrules_hide_all 

add include $devfsrules_unhide_basic 

add include $devfsrules_unhide_login 

add include $devfsrules_jail 

add path pf unhide 

add path pflog unhide 


and applying it to the jail: 


rootOjailhost:~ # service devfs restart 
rootOjailhost:~ # iocage stop vnetcage 

root@jailhost:~ # iocage set devfs_ruleset=501 vnetcage 
devfs_ruleset: 4 -> 501 

rootOjailhost:~ # iocage console -f vnetcage 

root@vnetcage:~ # Is /dev/pf 
/dev/pf 

rootOvnetcage:~ # logout 
rootOjailhost:~ # 

Note: Due to a bug in iocage(8), the configured devfs(8) ruleset is removed 
from devfs(8) every time the jail is stopped. This will probably get fixed in a 
future release; in the meantime there's a patch available 
(https://github.eom/iocage/iocage/pull/1106) that addresses the issue. 
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Test using a minimal firewall configuration—check if blocking a specific IP address 
works and if state is created like expected: 


rootSjailhost:~ # iocage console -f vnetcage 

rootOvnetcage:~ # echo "set skip on loO" >/etc/pf.conf 

rootOvnetcage:~ # echo "block quick to 1.1.1.1" »/etc/pf.conf 

rootOvnetcage:~ # echo "pass" »/etc/pf.conf 

root@vnetcage:~ # service pf enable 

pf enabled in /etc/rc.conf 

rootOvnetcage:~ # service pf start 

Enabling pf. 

root@vnetcage:~ # service pf start 
rootOvnetcage:~ # ping -cl 8.8.8.8 
PING 8.8.8.8 (8.8.8.8): 56 data bytes 

64 bytes from 8.8.8.8: icmp_seq=0 ttl=56 time=16.475 ms 
- 8.8.8.8 ping statistics - 

1 packets transmitted, 1 packets received, 0.0% packet loss 

round-trip min/avg/max/stddev = 16.475/16.475/16.475/0.000 ms 

root@vnetcage:~ # pfctl -s state 

all icmp 10.1.1.2:39231 -> 8.8.8.8:39231 0:0 

rootOvnetcage:~ # ping -cl 1.1.1.1 

PING 1.1.1.1 (1.1.1.1): 56 data bytes 

ping: sendto: Permission denied 

- 1.1.1.1 ping statistics - 

1 packets transmitted, 0 packets received, 100.0% packet loss 
root@vnetcage:~ # fetch -q -o - http://canhazip.com 
93.104.68.83 

rootOvnetcage:~ # pfctl -s state 

all icmp 10.1.1.2:39231 -> 8.8.8.8:39231 0:0 

all udp 10.1.1.2:32682 -> 8.8.8.8:53 MULTIPLE:SINGLE 

all tcp 10.1.1.2:58192 -> 104.16.223.38:80 FIN_WAIT_2:FIN_WAIT_2 

rootOvnetcage:~ # logout 

root®j ailhost:~ # 

Note: Jails/VMs on the same bridge can set/steal IPs and change MAC addresses, so 
this level of protection is only sufficient to prevent unwanted./accidental traffic from 
happening. For tighter security, it's possible to enable Layer 2 filtering using ipfw(8) 
on the jailhost and insert an additional bridge between the host bridge and the 
jail's epair interface. The details of such a setup are beyond the scope of this article. 



Jails and VMs are virtualization technologies that add a lot of flexibility and agility 
to the provisioning of resources. 

VXLAN (Virtual extensible LAN interface) is a tunneling protocol that aims to 
decouple the virtual network from the underlying physical network and thereby 
ease automation and orchestration. It does so by encapsulating Layer 2 Ethernet 
frames into Layer 3 IP/UDP packets. One can think of VXLAN as VLAN for multi- 
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tenant data centers. And just like VLAN uses a unique VLAN tag, VXLAN uses a 
VXLAN Network Identifier (VNI), a 24-bit value in the VXLAN header, to distin¬ 
guish network segments. 

VXLAN Example Overview 

The example setup consists of three hosts on the same LAN, one gateway host 
(LAN IP 192.168.0.1) and two jailhosts ("jailhost-a" and "jailhost-b") with LAN IP 
addresses 192.168.0.10 and 192.168.0.20. The jailhosts are hosting VNET jails 
and VMs; host "jailhost-b" also hosts a plain jail. 

The three hosts are connected over two VXLANs (VXLAN ids 111 and 222); the 
gateway host provides access to the internet over a dedicated uplink using NAT. 
Jails and VMs get IP addresses on both VXLANs (networks 10.0.111.0/24 and 
10.0.222.0/24). 

In this example, many configurations will be applied by rebooting the machines 
involved. Even though this is mostly done for the sake of brevity, it also reboot 
tests the setup as a side-effect—something that needs to be done anyway. 
Everything described can also be accomplished without rebooting though. 
Network diagram: 

.-( )-. 

( internet ) 

'-( 


28 


Gateway 


I emO I 


.I vxlanllll vxlan222 |. 
I eml | 


-1- 

I 

> - . - < 

.>1 IP Switch |<. 


/ 


jailhost-a 

/ 

I igbO I 

...I vxlanlll| vxlan222 I. 


/ 


\ 


/ \ 

I switch222 | I switchlll | 


I 


I 


I I 

_I_ _I_ 

I vnetjail-a I I guest-a I 


\ jailhost-b 

\ 

I emO | 

..| vxlanlll| vxlan222 I.... 

I__I__I 

/ \ 

' \ 


I switchlll | | switch222 I 


I I 


I 


- -| | 

_I_ _I_ 

I vnetjail-b I I guest-b | 


>| plainjail-b | 


FreeBSD Journal 

















































































































































The example uses multicast mode. VXLAN can also be configured in unicast 
mode (see vxlan(4) for details). 

Gateway Configuration 

The gateway host uses a simple ipfw(8)/natd(8) configuration and two vxlan(4) 
interfaces. It is assumed that the uplink is already configured: 

Firewall, NAT and IP forwarding: 


rootOgateway:~ # service ipfw enable 
ipfw enabled in /etc/rc.conf 
rootOgateway:~ # sysrc firewall_type=open 
firewall_type: UNKNOWN -> open 
root@gateway:~ # service natd enable 
natd enabled in /etc/rc.conf 
rootOgateway:~ # sysrc natd_interface=emO 
natd_interface: -> emO 

rootOgateway:~ # sysrc gateway_enable=YES 
gateway_enable: NO -> YES 


VXLAN interfaces: 


rootOgateway:~ # sysrc cloned_interfaces+="vxlanlll vxlan222" 
cloned_interfaces: -> vxlanlll vxlan222 

rootOgateway:~ # sysrc ifconfig_vxlanlll="inet 10.0.111.1/24 mtu 1450" 
ifconfig_vxlanlll: -> inet 10.0.111.1/24 mtu 1450 

root@gateway:~ # sysrc create_args_vxlanlll="vxlanid 111 vxlanlocal\ 
192.168.0.1 vxlandev eml vxlangroup 239.0.0.111" 
create_args_vxlanlll: -> vxlanid 111 vxlanlocal 192.168.0.1 

vxlandev eml vxlangroup 239.0.0.111 

root@gateway:~ # sysrc ifconfig_vxlan222="inet 10.0.222.1/24 mtu 1450" 
ifconfig_vxlan222: -> inet 10.0.222.1/24 mtu 1450 

root@gateway:~ # sysrc create_args_vxlan222="vxlanid 222 vxlanlocal\ 
192.168.0.1 vxlandev eml vxlangroup 239.0.0.222" 
create_args_vxlan222: -> vxlanid 222 vxlanlocal 192.168.0.1 

vxlandev eml vxlangroup 239.0.0.222 


Static routes to VXLAN multicast addresses: 


root@gateway:~ # sysrc static_routes+="vxlanlll vxlan222" 
static_routes: -> vxlanlll vxlan222 

root^gateway:~ # sysrc route_vxlanlll="239.0.0.111/32 -interface eml" 
route_vxlanlll: -> 239.0.0.111/32 -interface eml 

root@gateway:~ # sysrc route_vxlan222="239.0.0.222/32 -interface eml" 
route_vxlan222: -> 239.0.0.222/32 -interface eml 

rootQgateway:~ # reboot 


Jailhost-a 

"jailhost-a" hosts one VM and one VNET jail, connected over switches (bridge 
interfaces) to the VXLAN interfaces, which in turn use the physical interface igbO to 
transmit the encapsulated traffic over the LAN. 
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Network Configuration (jailhost-a) 

Create VXLAN interfaces: 


root@jailhost-a:~ # sysrc cloned_interfaces+="vxlanlll vxlan222" 
cloned_interfaces: -> vxlanlll vxlan222 

root@jailhost-a:~ # sysrc ifconfig_vxlanlll="inet 10.0.111.10/24 mtu 1450" 
ifconfig_vxlanlll: -> inet 10.0.111.10/24 mtu 1450 

root@jailhost-a:~ # sysrc create_args_vxlanlll="vxlanid 111 vxlanlocal\ 
192.168.0.10 vxlandev igbO vxlangroup 239.0.0.111" 
create_args_vxlanlll: -> vxlanid 111 vxlanlocal 192.168.0.10 

vxlandev igbO vxlangroup 239.0.0.111 

root@jailhost-a:~ # sysrc ifconfig_vxlan222="inet 10.0.222.10/24 mtu 1450" 
ifconfig_vxlan222: -> inet 10.0.222.10/24 mtu 1450 

root@jailhost-a:~ # sysrc create_args_vxlan222="vxlanid 222 vxlanlocal\ 
192.168.0.10 vxlandev igbO vxlangroup 239.0.0.222" 
create_args_vxlan222: -> vxlanid 222 vxlanlocal 192.168.0.10 

vxlandev igbO vxlangroup 239.0.0.222 


Set static routes for multicast traffic (technically not needed if the default route 
goes through the same interface): 


root@jailhost-a:~ # sysrc static_routes+="vxlanlll vxlan222" 
static_routes: -> vxlanlll vxlan222 

root@jailhost-a:~ # sysrc route_vxlanlll="239.0.0.111/32 -interface igbO" 
route_vxlanlll: -> 239.0.0.111/32 -interface igbO 

root@jailhost-a:~ # sysrc route_vxlan222="239.0.0.222/32 -interface igbO" 
route_vxlan222: -> 239.0.0.222/32 -interface igbO 

root@jailhost-a:~ # reboot 


Create switches (bridge interfaces) to connect jails/VMs to and add the respec¬ 
tive VXLAN interfaces to them: 


root@jailhost-a:~ # sysrc cloned_interfaces+="bridge0 bridgel" 
cloned_interfaces: vxlanlll vxlan222 -> vxlanlll vxlan222 bridgeO bridgel 
root@jailhost-a:~ # sysrc ifconfig_bridgeO_name="switchlll" 
ifconfig_bridgeO_name: -> switchlll 

root@jailhost-a:~ # sysrc \ 

ifconfig_switchlll="inet 10.0.111.12/32 addm vxlanlll" 
ifconfig_switchill: -> inet 10.0.111.12/32 addm vxlanlll 

root@jailhost-a:~ # sysrc ifconfig_bridgel_name="switch222" 
ifconfig_bridgel_name: -> switch222 

root@jailhost-a:~ # sysrc \ 

ifconfig_switch222="inet 10.0.222.12/32 addm vxlan222" 
ifconfig_switch222: -> inet 10.0.222.12/32 addm vxlan222 

root@jailhost-a:~ # reboot 


VM Configuration (jailhost-a) 

Install sysutils/vm-bhyve and create a switch of type "manual" (not managed by 
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vm-bhyve) that refers to the bridge interface (switch 111) created above: 


root@jailhost-a:~ # pkg install vm-bhyve 

root@jailhost-a:~ # service vm enable 
vm enabled in /etc/rc.conf 

root@jailhost-a:~ # sysrc vm_dir=zfs:zroot/vms 
vm_dir: -> zfs:zroot/vms 

root@jailhost-a:~ # zfs create zroot/vms 
root@jailhost-a:~ # vm init 

root@jailhost-a:~ # vm switch create -t manual -b switchlll switchlll 
root@jailhost-a:~ # vm switch list 

NAME TYPE IFACE ADDRESS PRIVATE MTU VLAN PORTS 

switchlll manual switchlll n/a no n/a n/a n/a 


Download FreeBSD ISO and install a VM called "guest-a"; start new VM on boot: 


root@jailhost-a:~ # vm iso https://download.freebsd.org/ftp/releases/X 
ISO-IMAGES/12.I/FreeBSD-12.l-RELEASE-amd64-bootonly.iso 
root@jailhost-a:~ # vm create guest-a 

root@jailhost-a:~ # vm add -d network -s switchlll guest-a 
root@jailhost-a:~ # vm install \ 

guest-a FreeBSD-12.l-RELEASE-amd64-bootonly.iso 
root@jailhost-a:~ # vm console guest-a 

... (set IP of vtnetl to 10.0.111.13/24, set default gw to 10.0.111.1) 
root@jailhost-a:~ # sysrc vm_list+="guest-a" 
root@jailhost-a:~ # vm stop guest-a 

Start the VM and test connectivity (note how public traffic is sent over VXLAN 
111 and NATted at the gateway host, as the gateway host is configured to be the 
default gateway): 


root@jailhost-a:~ # vm start guest-a 

root@jailhost-a:~ # vm console guest-a 

root@guest-a:~ # ping -c 3 10.0.111.1 

PING 10.0.111.1 (10.0.111.1): 56 data bytes 

64 bytes from 10.0.111.1: icmp_seq=0 ttl=64 time=1.545 ms 

64 bytes from 10.0.111.1: icmp_seq=l ttl=64 time=0.793 ms 

64 bytes from 10.0.111.1: icmp_seq=2 ttl=64 time=0.790 ms 

- 10.0.111.1 ping statistics - 

3 packets transmitted, 3 packets received, 0.0% packet loss 
round-trip min/avg/max/stddev = 0.790/1.043/1.545/0.355 ms 
root@guest-a:~ # traceroute www.freebsd.org 
1 10.0.111.1 (10.0.111.1) 1.251 ms 1.421 ms 1.070 ms 

root@guest-a:~ # logout 
$ ~D 
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Jail Configuration (jailhost-a) 

Install sysutils/iocage: 


root@jailhost-a:~ # pkg install py36-iocage 
root@jailhost-a:~ # iocage activate zroot 
ZFS pool 'zroot' successfully activated. 
root@jailhost-a:~ # mount -t fdescfs null /dev/fd 
root@jailhost-a:~ # service iocage enable 
iocage enabled in /etc/rc.conf 

Note: You might want to mount fdescfs(5) permanently by altering /etc/fstab. 

Create jail named "vnetjail-a" on VXLAN 222 (configured to be started at boot 
time, so it will start immediately after creation): 


root@jailhost-a:~ # iocage create -n vnetjail-a -r 12.1-RELEASE \ 
interfaces="vnetO:switch222" ip4_addr="vnet0110.0.222.13/24" \ 
boot=l vnet_default_interface="switch222" defaultrouter="10.0.222.1" \ 
vnet=on 

vnetjail-a successfully created! 


Enter the jail and test connectivity: 


root@jailhost-a:~ # iocage console vnetjail-a 
root@vnetjail-a:~ # traceroute -n 141.1.1.1 

traceroute to 141.1.1.1 (141.1.1.1), 64 hops max, 40 byte packets 

1 10.0.111.1 (10.0.111.1) 1.353 ms 0.687 ms 1.045 ms 

2 many more interesting hosts... 

root@vnetjail-a:~ # fetch -q -o - http://canhazip.com 
(your public IP here) 
rootOvnetjail-a:~ # logout 


Finally, test if everything comes up correctly on reboot: 


root@jailhost-a:~ # reboot 


Jailhost-b 

The setup of "jailhost-b" is similar to that of "jailhost-a", but with networks 
reversed (VM on VXLAN 222, jails on VXLAN 111). In addition, it hosts a plain jail that 
uses an alias address directly on the VXLAN interface (vxlanl 11), which doesn't use 
that network as its default gateway. In a production setup, this jail could be used to 
contain a supporting service provided by the underlying jailhost. 

Network Configuration (jailhost-b) 

Create VXLAN interfaces: 
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root@jailhost-b:~ # sysrc cloned_interfaces+="vxlanlll vxlan222" 
cloned_interfaces: -> vxlanlll vxlan222 

root@jailhost-b:~ # sysrc ifconfig_vxlanlll="inet 10.0.111.20/24 mtu 1450" 
ifconfig_vxlanlll: -> inet 10.0.111.20/24 mtu 1450 

root@jailhost-b:~ # sysrc create_args_vxlanlll="vxlanid 111 vxlanlocal\ 
192.168.0.20 vxlandev emO vxlangroup 239.0.0.111" 
create_args_vxlanlll: -> vxlanid 111 vxlanlocal 192.168.0.20 

vxlandev emO vxlangroup 239.0.0.111 

root@jailhost-b:~ # sysrc ifconfig_vxlan222="inet 10.0.222.20/24 mtu 1450" 
ifconfig_vxlan222: -> inet 10.0.222.20/24 mtu 1450 

root@jailhost-b:~ # sysrc create_args_vxlan222="vxlanid 222 vxlanlocal\ 
192.168.0.20 vxlandev emO vxlangroup 239.0.0.222" 
create_args_vxlan222: -> vxlanid 222 vxlanlocal 192.168.0.20 

vxlandev emO vxlangroup 239.0.0.222 


Set static routes for multicast traffic (technically not needed if the default route 
goes through the same interface): 


root@jailhost-b:~ # sysrc static_routes+="vxlanlll vxlan222" 
static_routes: -> vxlanlll vxlan222 

root@jailhost-b:~ # sysrc route_vxlanlll="239.0.0.111/32 -interface emO" 
route_vxlanlll: -> 239.0.0.111/32 -interface emO 

root@jailhost-b:~ # sysrc route_vxlan222="239.0.0.222/32 -interface emO" 
route_vxlan222: -> 239.0.0.222/32 -interface emO 

root@jailhost-b:~ # reboot 


Plain Jail Configuration (jailhost-b) 

Install sysutils/iocage: 


root@jailhost-b:~ # pkg install py36-iocage 

root(9jailhost-b: ~ # iocage activate zroot 
ZFS pool 'zroot' successfully activated, 
root®jailhost-b:~ # mount -t fdescfs null /dev/fd 
rootQjailhost-b:~ # service iocage enable 
iocage enabled in /etc/rc.conf 

Note: You might want to mount fdescfs(5) permanently by altering /etc/fstab. 


Create a plain ( non-VNET) jail on the alias address 10.0.111.21. It's configured to 
run at boot time, so it starts immediately after creation: 


root®jailhost-b:~ # iocage create -n plainjail-b \ 

-r 12.1-RELEASE ip4_addr="vxlanlll|10.0.Ill.21/32" \ 
allow_raw_sockets=l boot=l 
plainjail-b successfully created! 
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Run jail and test connectivity: 


root@jailh.ost-b: ~ # iocage console -f plainjail-b 

root@plainjail-b:~ # ping -c 3 10.0.111.1 
PING 10.0.111.1 (10.0.111.1): 56 data bytes 
64 bytes from 10.0.111.1: icmp_seq=0 ttl=64 time=0.811 ms 
64 bytes from 10.0.111.1: icmp_seq=l ttl=64 time=0.816 ms 
64 bytes from 10.0.111.1: icmp_seq=2 ttl=64 time=0.810 ms 

- 10.0.111.1 ping statistics - 

3 packets transmitted, 3 packets received, 0.0°/ 0 packet loss 
round-trip min/avg/max/stddev = 0.810/0.812/0.816/0.003 ms 
root@plainjail-b:~ # ping -c 3 10.0.111.10 
PING 10.0.111.10 (10.0.111.10): 56 data bytes 
64 bytes from 10.0.111.10: icmp_seq=0 ttl=64 time=0.716 ms 
64 bytes from 10.0.111.10: icmp_seq=l ttl=64 time=0.297 ms 
64 bytes from 10.0.111.10: icmp_seq=2 ttl=64 time=0.319 ms 

- 10.0.111.10 ping statistics - 

3 packets transmitted, 3 packets received, 0.0% packet loss 
round-trip min/avg/max/stddev = 0.297/0.444/0.716/0.193 ms 
root@plainjail-b:~ # logout 
root@jailhost-b:~ # 

Note: The way "plainjail-b" is configured, its public traffic won't get sent over 
VXLAN and NATted by the gateway host. That's intentional in this setup; otherwise 
a VNETjail would've been used. 


Network Switch Setup (jailhost-b) 

Create switched (bridge interfaces) to connect jails/VMs to and add the respective 
VXLAN interfaces to them: 


root@jailhost-b:~ # sysrc cloned_interfaces+="bridge0 bridgel" 
cloned_interfaces: vxlanlll vxlan222 -> vxlanlll vxlan222 bridgeO bridgel 
root@jailhost-b:~ # sysrc ifconfig_bridgeO_name="switchlll" 
ifconfig_bridgeO_name: -> switchlll 

root@jailhost-b:~ # sysrc \ 

ifconfig_switchlll="inet 10.0.111.22/32 addm vxlanlll" 
ifconfig_switchlll: -> inet 10.0.111.22/32 addm vxlanlll 

root@jailhost-b:~ # sysrc ifconfig_bridgel_name="switch222" 
ifconfig_bridgel_name: -> switch222 

root@jailhost-b:~ # sysrc \ 

ifconfig_switch222="inet 10.0.222.22/32 addm vxlan222" 
ifconfig_switch222: -> inet 10.0.222.22/32 addm vxlan222 

root@jailhost-b:~ # reboot 


VNET Jail Configuration (jailhost-b) 

Create jail named "vnetjail-b" on VXLAN 111 (configured to be started at boot time, 
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so it'll start immediately after creation): 


root@jailhost-b:~ # iocage create -n vnetjail-b -r 12.1-RELEASE \ 
interfaces="vnetO:switchlll" ip4_addr="vnet0|10.0.Ill.23/24" \ 
boot=l vnet_default_interface="switchlll" defaultrouter="10.0.111.1" \ 
vnet=on 

vnetjail-b successfully created! 


Enter the jail and test connectivity: 


root@jailhost-b:~ # iocage console vnetjail-b 
rootOvnetjail-b:~ # traceroute -n 141.1.1.1 

traceroute to 141.1.1.1 (141.1.1.1), 64 hops max, 40 byte packets 

1 10.0.111.1 (10.0.111.1) 1.353 ms 0.687 ms 1.045 ms 

2 many more interesting hosts... 

root@vnetjail-b:~ # logout 

VM Configuration (jailhost-b) 

Install sysutils/vm-bhyve and create a switch of type "manual" (not managed by 
vm-bhyve) that refers to the bridge interface (switchl 11) created above: 

root@jailhost-b:~ # pkg install vm-bhyve 

root@jailhost-b:~ # service vm enable 
vm enabled in /etc/rc.conf 

root@jailhost-b:~ # sysrc vm_dir=zfs:zroot/vms 
vm_dir: -> zfs:zroot/vms 

root@jailhost-b:~ # zfs create zroot/vms 
root@jailhost-b:~ # vm init 

root@jailhost-b:~ # vm switch create -t manual 
root@jailhost-b:~ # vm switch list 
NAME TYPE IFACE ADDRESS PRIVATE 

switch222 manual switch222 n/a no 


Download FreeBSD ISO and install a VM called "guest-b"; start new VM on boot: 


-b switch222 switch222 

MTU VLAN PORTS 
n/a n/a n/a 


root@jailhost-b:~ # vm iso https://download.freebsd.org/ftp/releases/\ 
ISO-IMAGES/12.I/FreeBSD-12.l-RELEASE-amd64-bootonly.iso 
root@jailhost-b:~ # vm create guest-b 

root@jailhost-b:~ # vm add -d network -s switch222 guest-b 
root@jailhost-b:~ # vm install \ 

guest-b FreeBSD-12.l-RELEASE-amd64-bootonly.iso 
root@jailhost-b:~ # vm console guest-b 

... (set IP of vtnetl to 10.0.222.23/24, set default gw to 10.0.222.1) 
root@jailhost-b:~ # sysrc vm_list+="guest-b" 
root@jailhost-b:~ # vm stop guest-b 
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Start the VM and test connectivity (note how public traffic is sent over VXLAN 
222 and NATted at the gateway host, as the gateway host is configured to be the 
default gateway): 


root@jailhost-b:~ # vm start guest-b 
root@jailhost-b:~ # vm console guest-b 
root@guest-b:~ # traceroute www.freebsd.org 
1 10.0.111.1 (10.0.111.1) 1.251 ms 1.421 ms 1.070 ms 

root@guest-b:~ # logout 
$ ~D 


Finally, test if everything comes up correctly on reboot: 

root@jailhost-b:~ # reboot 


VXLAN Multicast Troubleshooting 

There are various tools that can help to troubleshoot your VXLAN setup. 
Use ifconfig(8) to make sure the interface is actually configured correctly: 


root@jailhost-b:~ # ifconfig vxlanlll 

vxlanll1: flags=8943<UP,BROADCAST,RUNNING,PR0MISC,SIMPLEX,MULTICAST 
metric 0 mtu 1450 

options=80000<LINKSTATE> 
ether 58:9c:fc:10:ff:fe 

inet 10.0.111.20 netmask OxffffffOO broadcast 10.0.111.255 
inet 10.0.111.21 netmask Oxffffffff broadcast 10.0.111.21 
groups: vxlan 

vxlan vni 111 local 192.168.0.20:4789 group 239.0.0.111:4789 
media: Ethernet autoselect (autoselect <full-duplex>) 
status: active 

nd6 options=29<PERF0RMNUD,IFDISABLED,AUT0_LINKL0CAL> 

Note: The MTU is set to 1450 (as VXLAN uses 50 bytes of header information). If 
supported by all components, it's recommended to configure jumbo frames on 
your network. 

Use sockstat(l) to make sure that the host is actually listening on the standard 
VXLAN port 4789: 


root@jailhost-b:~ # sockstat -14 | grep 4789 
? ? ? ? udp4 *:4789 *:* 


Use ifmcstat(8) to verify that the multicast configuration looks reasonable: 


root@jailhost-b:~ # ifmcstat -i emO 
emO: 

inet 192.168.0.20 
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igmpv3 rv 2 qi 125 qri 100 uri 3 

group 239.0.0.222 mode exclude 

mcast-macaddr 01:00:5e:00:00:de 
group 239.0.0.111 mode exclude 

mcast-macaddr 01:00:5e:00:00:6f 
group 224.0.0.1 mode exclude 

mcast-macaddr 01:00:5e:00:00:01 


Use tcpdump(l) to inspect VXLAN traffic: 


root@jailhost-b:~ # tcpdump -vni emO -f "udp && port 4789" 

tcpdump: listening on emO, link-type EN10MB (Ethernet), capture size\ 

262144 bytes 

14:24:13.288186 IP (tos 0x0, ttl 64, id 9404, offset 0, flags [none],\ 
proto UDP (17), length 78) 

192.168.0.20.17657 > 239.0.0.111.4789: VXLAN, flags [I] (0x08), vni 111 
ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.0.111.1 tell\ 
10.0.111.20, length 28 


Check the ARP table using arp(8) (also within VMs): 


root@jailhost-b:~ # arp -an 
? (10.0.222.22) at 02:Oe:35:b7:6d:01 
? (10.0.111.22) at 02:Oe:35:b7:6d:00 
? (10.0.222.20) at 58:9c:fc:10:ff:81 
? (10.0.111.21) at 58:9c:fc:10:ff:fe 
? (10.0.111.20) at 58:9c:fc:10:ff:fe 
? (10.0.111.12) at 02:f7:16:28:83:00 
[ethernet] 


on switch222 permanent [bridge] 
on switchlll permanent [bridge] 
on vxlan222 permanent [ethernet] 
on vxlanlll permanent [ethernet] 
on vxlanlll permanent [ethernet] 
on vxlanlll expires in 1160 seconds\ 


Check VXLAN forwarding tables using sysctl(8)\ 


root@jailhost-b:~ # sysctl net.link.vxlan.Ill.ftable.dump 
net.link.vxlan.Ill.ftable.dump: 


D 0x01 02:F7:16:28:83:00 
D 0x01 58:9C:FC:10:FF:E9 
D 0x01 00:BD:0B:07:F7:01 
D 0x01 58:9C:FC:03:25:35 


192.168.0.10 00002343 
192.168.0.10 00002378 
192.168.0.10 00002299 
192.168.0.10 00002347 


root@jailhost-b:~ # sysctl net.link.vxlan.222.ftable.dump 
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Flush ARP and VXLAN forwarding tables: 


root@jailhost-b:~ # arp -ad 
10.0.111.12 (10.0.111.12) deleted 
10.0.111.10 (10.0.111.10) deleted 
192.168.0.10 (192.168.0.10) deleted 
root@jailhost-b:~ # ifconfig vxlanlll vxlanflush 
root@jailhost-b:~ # ifconfig vxlan222 vxlanflush 


If the environment permits, temporarily disable host firewalls while trou¬ 
bleshooting to make sure they're not interfering with traffic. 

: Conclusion and Further Reading 

The intent of this article was to show various ways to configure the network of 
FreeBSD's virtualization features based on examples. The choice to use third 
party tools from ports/packages was made consciously, as many users will try 
these tools in practice to get started. Even if specific tools won't make it to a 
production setup, using them while prototyping makes it easier to experiment 
and perceive what's configured before building a leaner solution. 

The examples shown are by no means exhaustive, but they should help the 
reader to get started and figure out which kind of setup might meet their 
• specific requirements. It's recommended to do some further reading to 

fully understand the available options and their implications. • 

Some good so urces of topic inform a tion 
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Michael W Lucas — FreeBSD Mastery: Jails (https://mwl.io/nonfiction/os#fmjail) 

John Nielsen — Using VXLAN to network virtual machines, jails, and other fun things on 
FreeBSD (https://www.bsdcan.org/2016/schedule/events/715.en.html) 

Slides (https://www.bsdcan.org/2016/schedule/attachments/341 VXLAN BSDCan2016.pdf) 

Video (https://www.youtube.com/watch7v- INe TgF3MQ) 
iocage: A FreeBSD Jail Manager (https://iocaae.readthedocs.io) 

vm-bhyve: Shell based, minimal dependency bhyve manager (https://github.com/churchers/vm-bhvve) 
pot: another container framework for FreeBSD, based on jails, ZFS and pf (https://github.com/pizzamig/pot) 
FreeBSD as a Host with bhyve (https://www.freebsd.orq/doc/handbook/virtualization-host-bhwe.html) , 
covers setting up bhyve by directly using LAN addresses. 

FreeBSD ifconfig(8) man page (https://www.freebsd.orq/cqi/man.cqi?query-ifconfiq&sektion-8) 

FreeBSD bridge(4) man page (https://www.freebsd.org/cgi/man.cgi?querv=bridge&sektion=4) 

FreeBSD vxlan(4) man page (https://www.freebsd.orq/cqi/man.cqi?querv=vxlan&sektion=4) 


Growing up in the Bavarian countryside, Michael Gmelin started using FreeBSD 
at a rural userfriendly.org-style ISP in the 1990s. Spending most of the aughts 
developing software to automate ISP and network operator processes, he start¬ 
ed contributing to the Project close to the end of the decade. While he moved 
on to build payment systems in the emerging fintech industry, he finally became 
a FreeBSD committer in 2014. Having written software in many different lan¬ 
guages, he recently developed an affection for Rust. In his spare time, he likes to 
play classic video games and run demoscene productions. When away from the 
keyboard, he enjoys (making) music, cycling, woodworking, and cooking. Perl 
will always have a place in his heart. 
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The FreeBSD Journal Editorial Board suggested that some of 
the outstanding interviews from the BSD Now series might 
be of interest to readers as they reflect the state of technolo¬ 
gy when the interview took place. Here, we've transcribed 
and excerpted from a 2018 interview with Kirk McKusick by 
Allan Jude and Benedict Reuschling. 



OYouTube Search Allan Jude, Kirk McKusick, and Benedict Reuschling 


t 



Benedict: In this episode of BSD Now, we're actually enjoying the holidays a 
little bit, so we have picked something from the shelf. We sat down at 
BSDCan 2018, way back when, to interview Kirk McKusick about various 
topics ranging about the early years of BSD, Unix, the continuing work of 
UFS that he is working on, the governance of FreeBSD, and more. 

We're joined today by Kirk McKusick, who is one of the early committers 
to the FreeBSD Project and a principle of the BSD project at the University of 
California at Berkeley. So, welcome. What we want to know first is how you 
got involved with Unix and BSD. 

• Kirk: I was office mates with Bill Joy, and it was Bill Joy's project 
originally. It was impossible not to be involved because there were 
four desks and four people. There was one phone which was for 


40 


FreeBSD Journal 












everybody to use, and if the phone would ring, if I happened to be near 
it. I'd just pick it up and hand it to Bill because 99% of the time it was for 
Bill. So why even bother answering it if you were anyone else? 

Benedict: During your early involvement with BSD, did you have any idea that it 
would become what it is today? 

• Kirk: Not a clue. I'm terrible about predicting the future anyway. But to 
me it just was this thing where Bill had some programs and you couldn't 
be around Bill without getting sort of sucked into doing things. And he'd 
say, "Oh, could you just do this one little thing?" For example, we had a 
Pascal compiler, and it compiled into byte codes and then the interpreter 
would interpret them, and that interpreter had been written in Assembly 
language. 

And then we got a VAX and Bill said, "Could you just port this Pascal 
interpreter over to the VAX? There's the MOVE instruction on the PDP-11, 
and there's the MOVE instruction on the VAX. It will just be simple trans¬ 
formation." Ha-ha-ha. And three weeks later, something sort of popped 
out. But he just made everything sound like it was going to be so simple 
and so easy, and we would just do it. 

The distribution was just Bill passing out tapes. He would go to confer¬ 
ences and talk up BSD, and people would get all excited and would get 
the latest BSD distribution. And code would be coming back, and it just 
sort of bubbled up and bubbled up and bubbled up, and at the time, it 
was like one big software project. Lots of people wrote programs and 
passed them around so that they would become what they became. No 
chance I would have guessed that. 

Allan: Yes. 

Benedict: Maybe that motivated other people to contribute because they had no 
other chance of escaping that during his presentations? 

• Kirk: Yeah, the thing is that Bill would go out and he would talk with 
people: what are you doing, blah-blah-blah? The early EX editor started 
from the EM editor written by Professor George Coulouris at Queen 
Mary's College, London. And Bill's attitude was always, why come up with 
something good when you can use something that already exists and is 
better. So, he would find something else that someone else had done 
and then just embellish it to be whatever he wanted it to be. 

And it was actually not Bill, but Keith Bostic who would say there was 
no end to the good that you can do in the world by giving credit to 
other people. And in particular, when we were trying to rewrite AT&T's 
programs, he would get other people to write CAT or LS or whatever it 
was, and contribute it to Berkeley and he put their name all over it, and 
9 times out of 10 would rewrite 80% of what was in there. 

Benedict: [laughs] Okay. 

Allan: But yeah, that's a big part of the BSD philosophy, that giving credit is the 
important thing. 
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• Kirk: Right, and by doing so—as Keith put it at one point where AT&T 
was bickering over whether this was the original or not, he'd say, I 
always have someone else to blame. You need to go talk to so-and-so 
because they wrote that! 

Allan: There's definitely a lot of history in the files. 

• Kirk: Correct. 

Allan: When you look at the number of people who have touched any given 
file, it's just like wow! 

• Kirk: Yes, some of them are truly amazing. There's a file that I 
thought I had written. When I went and looked, I think I found four or 
five lines that I had put in and which probably are part of the copyright 
message. 

Benedict: And it's also the care and the maintenance that people have put into 
these programs and passed along to make it still work on your hardware or 
with your environments or your file formats. 

Allan: Yes. Entire architectures like RISC-V are brand-new but are going to run 
all this old code with only a little bit of help. 

• Kirk: Yes, exactly. Care and feeding is such a huge amount of what 
keeps a system viable. Of course, it is what's new, what's exciting, that 
attracts interest. 

Documentation is an area that is important but not exciting for most 
programmers. I remember now many years ago someone came up and 
said, "Oh, I'm interested in FreeBSD. And I'm just a person who writes 
documentation. I can't write any code. Is there anything I can do?" And 
I said, "Well, as chance would have it, we have these Manual pages." 
Benedict: We've been waiting for you. 

• Kirk: Right, I think documentation is a big part of what has made 
FreeBSD successful, and the thing that made the documentation suc¬ 
cessful is that we don't have this whole social hierarchy. 

Benedict: Yeah. 

• Kirk: If you are a committer, you get just as much right to vote for 
core whether you're committing to documentation, source, ports, sys¬ 
tem administration scripts, or whatever. 

Allan: Or even to be on the core team. 

Benedict: Yeah. 

Allan: It's not only source committers who are eligible to stand in the election. 

• Kirk: Correct. People realize that it's more than just code that's impor¬ 
tant in the FreeBSD Project. 

Benedict: Yeah, and it's also the tendency that people who got appreciation for 
the work they are doing are doing even more work. So, documenters become 
ports committers or source committers, and do both bits. 

Allan: That's how they sucked me in. I wrote some notes on the train ride here 
in 2013, and then in 2014, I was a documenter. In 2015, I was a source com¬ 
mitter. And then in 2016, I was on the core team. 
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• Kirk: Yeah, exactly ! 

Benedict: And especially at this conference, we identified new people with like, 
hmmm, that guy could be a perfect new source committer or let's just give him 
a bit. 

• Kirk: One of the things that gives FreeBSD or BSD a leg up over the 
Linux world is that we do understand you've got to keep bringing in new 
blood. Linux is aging out on the people at the top of the tree, and it's 
going to be very interesting to see how that carries over. 

When Linus Torvalds retires, who's going to step up to that? And how 
is that going to happen? 

Benedict: Yeah, and as the system grows, there's more code to maintain, and 
that's just too much for a single person or just a small group to do. And there 
are also people leaving of course. Life happens to every one of us, but as long 
as we bring in more people than are leaving, then the project is still healthy. 

• Kirk: There was a period in BSD where if I went on vacation for two 
weeks, things couldn't get fixed. And my goal was if I get hit by a bus, 
BSD needs to be able to keep going. 

Benedict: Yeah, the Project is still live. 

• Kirk: And I'm way past the point where I'm a critical cog. 

Allan: I guess that leads into our next question, which is, what do you think is 
the biggest accomplishment of the FreeBSD Project over the last 25 years? 

• Kirk: Well, that's my governance talk, right? I think that we have 
evolved a system that allows it to continue moving forward, bringing in 
new people, and letting those people rise. There was a study that was 
done quite a few years ago now, and it was by people in economics at 
MIT and at the University of Toulouse in southern France. And they just 
studied source code commit logs to see who's in charge. 

And if you look at Linux, for example, it's been Linus Torvalds and his 
lieutenants, who are a fairly static set—throughout the lifetime of Linux. 
So, they've had one leadership. Most open-source projects go dark within 
about five years, where dark is defined as no commits for more than a 
year. Usually the person who started the project goes away. Occasionally 
they manage one leadership rollover but then it usually keels over. And 
only a couple of projects—Apache and FreeBSD and just one other—have 
had three or more successful changes in leadership. I think we're on our 
fifth, if you count BSD at Berkeley as one of them. So, four leadership 
changes in FreeBSD. 

Allan: Or can you count core teams in FreeBSD? 

• Kirk: Look at the people that make up the core team. The Jordan 
Flubbard era. There was a Robert Watson era. There's the current set of 
people. There have been people who have moved in and many of them 
are still sort of kicking around but aren't really leading at this point. 

Benedict: But the ability—the change of leadership is there— 

• Kirk: Right. 
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Benedict: —and the ability to continue on after that successfully. 

• Kirk: The committer population has continued to evolve, and our aver¬ 
age and median committer age is still under 40 after 25 years, though it 
has crept up. It started in the low 30s, and it's now high 30s. I think it's up 
to 40 now. If I actually chop off every committer who's over 60, I think we 
drop to around 38 as the average committer age. We just need to get rid 
of us old geezers. There's something like 12 of us. 

Benedict: I guess it's also a test of the durability of the Project. And I guess they 
lost a couple of people or the trust of some people that this is going to be a 
viable thing in the future? But, then once there, new people come and see that 
it's working and continue from that point on. 

• Kirk: Right. I mean one of the other things that I'm not sure I would call 
an accomplishment per se but the FreeBSD code base has a more studied 
advancement. The Linux Foundation is very proud of the fact that they've 
added a million lines of code to the Linux kernel in the space of a year or 
some short period like that. 

And to me that's an unmitigated disaster. More is not better. Windows 
learned this when they grew to where they had 50 million lines of code, 
and they couldn't stabilize it. Even given their enormous resources, they 
just couldn't. 

Allan: Right, too many moving parts. There's too much bulk. 

• Kirk: Right, and fixing bugs—with that many lines of code, you just 
can't—you never get the bugs fixed to the point where it becomes stable. 
Linux, just the kernel, is 20 million lines. The FreeBSD kernel is under 2 mil¬ 
lion lines and the entire FreeBSD distribution today is 16 million lines. But 
that includes the kernel and commands and libraries in the base system. 
Benedict: Yes, so what is your biggest surprise about what the project has 
become 25 years later. 

• Kirk: That it's still here and wildly successful. I mean given that the half- 
life of open-source software projects is five years, we've made it through 
five lifetimes. And if you count the original BSD in the total, from 1978 to 
now, it is 40 years. I gave a talk at the Usenix FAST conference. They asked 
me to give the keynote, and the keynote was on the evolution of the Fast 
File System (UFS) over 30 years. I started by saying if you had told me 30 
years ago that I was going to be giving a keynote about this software 30 
years later, I would have wondered what version of acid you would be on. 

Software has an even shorter half-life than projects have. So, three to 
five years is the lifetime of most software. To still have UFS running is 
quite unusual. Just as a lark, I took a disk image of a UFS filesystem from 
1982—which I had just saved as an image of the very first one I ever 
built—and I could actually read and write that on the current UFS imple¬ 
mentation. Though it did get a little bit cranky about some of the cylinder 
table mapping stuff. But it worked. 

Benedict: Yeah, there were some changes there. 
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• Kirk: Another interesting question is whether the much more modern 
Zettabyte File System (ZFS) is going to replace UFS. And the answer is no, 
because they each solve different problems. When you've got a lot of 
resources and huge filesystems, it's a no-brainer that ZFS is what you're going 
to use. But with an embedded system, you're just not going to use ZFS. You 
don't have the resources to run it. You need UFS, which is lean and mean. 
Allan: Yeah, and we're heading into an era where we're going to have billions 
more of these tiny devices, and they're all going to need a filesystem. 

• Kirk: Yeah. And it's not going to be ZFS. 

Allan: Yeah, and I'm sure you didn't expect to ever anticipate billions of installs of 
UFS. Or a billion volumes. 

• Kirk: Right, no. 

Allan: I don't think anybody thought there'd be a billion computers. 

Benedict: And you want to use something that's reliable and has been tested for so 
long and not just use a fresh development. Of course, they always happen but if 
you have something available, like UFS that's been driven and tested, it's a no- 
brainer to use that rather than just getting something off the ground. 

Allan: Well, it's not like UFS is stagnant either, right? It's got support and works 
well on modern SSDs. 

• Kirk: UFS needs care and feeding. When I first started at Berkeley, I didn't 
understand that concept. 

Benedict: You have to go back, yeah. 

• Kirk: So, thankfully, Rick Macklin did a lot of the initial work on NFS, and 
he more or less has been caring for it and feeding it ever since. And David 
Greenman originally, and then later Alan Cox has taken the VM system to 
places where you never would have guessed in a million years. And even 
Jeff Roberson does a lot of the UFS stuff these days. 

So, I can just sort of sit back and say, Oh! You ought to try doing this. 

Allan: Yes, yes. 

Benedict: So here history repeats itself. 

• Kirk: Yes. Being the "grand old man" can be kind of fun. You can have all 
the ideas and not have to do any of the work. 

Allan: You get to watch the project happen and get all the joy. 

Benedict: Yeah, enjoy the fruits of the labors. 

Allan: Yes, less stress. 

• Kirk: Someone asked me when am I going to run for core? It's like ha-ha- 
ha. No, never! I did that for 10 years at Berkeley. I've served my time. 

Allan: Yes. In looking back, what is your favorite FreeBSD-related memory? 

• Kirk: Oooh! Favorite FreeBSD-related memory? I mean there are so many 
different things that I can pick. Seeing things like the SMP finally ship and — 
well, not just ship in FreeBSD 5, but finally getting made to work in 
FreeBSD 6. Most of my memories are conferences like BSDCan, which kind 
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of started out as an ad hoc thing, and I go to it and it's like, wow, I get to 
see all my friends. 

Benedict: Yes, breaks records. 

• Kirk: Also things like EuroBSD. It started as EUUG, and then later it 
bifurcated into several groups but ultimately became the BSD conferences. 
And the fact that they came up with a model where they could really 
move around was great. Asia started out moving around, but the problem 
was everyone had to start from scratch, and so you'd get conferences that 
didn't work because they didn't really know what they were doing. 

A group of folks from around Europe created the NUUG Foundation 
that manages conference venue commitments, registration logistics, and 
money handling. That is what makes the conference moves between cities 
possible because the organizers need only focus on the logistics of their 
country and getting together the program. 

When it's always in the same place, then you don't get the local draw. 

So for example, in Tokyo, I can really only teach my class about once every 
four or five years because you have to build up to having enough new 
attendees. Whereas in Europe, I can pretty reliably just teach it year after 
year because we're in a different country and there's a whole bunch of 
new local people. They want to take it. And that works. 

Benedict: I like to see the faces behind all the commit emails, the community¬ 
building, the act of meeting people. 

Allan: Every time you get an email from them, you read it in their voice. 

• Kirk: Yes. 

Allan: And it makes a very big difference in the communication streaming 
because suddenly any ambiguity is slanted in a good way. 

Benedict: Yes, you touched on it a little bit before with the SMP work, but do you 
have a favorite milestone for the Project? 

• Kirk: Milestones that were interesting to me were things like getting 
soft updates into the system, and it's some of the technical things that I've 
been involved with directly or indirectly that are most memorable to me. 
But that is mostly because I've been directly involved in working on them. 
Allan: I know 64-bit inode numbers—that went on for some time. 

• Kirk: Yes, but that one finally came about because I found the person 
who was the most obstreperous and basically collared them into being 
involved in it. Once they were involved in it, they stopped being 
obstreperous and they got it done. I guess part of the thing that's sort of 
so interesting is it gets back to the interpersonal situations, where you 
get to know someone and figure out how they can be most effective. 

Allan: Yes, or how you can make the message make sense from their point of view. 

• Kirk: Or sometimes it's convincing other people that their point of view 
is correct. They may not be the best at getting that view out there. And 
one of the drawbacks about being the grand old man is that whatever I 
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say must be correct, even if it's complete bullshit. So, I try not to do stuff 
that's complete bullshit, obviously, but I will often find people who haven't 
been in the community long enough and are having trouble getting their 
ideas out, and just give it a push. 

Allan: Lend a bit of weight to the argument? 

• Kirk: Right. And when I say you should really listen to what Allan is say¬ 
ing, there are some good ideas here, doing it like this is the right way to 
go, people go, Oh, yeah, I guess I hadn't really thought about that. And 
then they come and engage you, or whoever, and once engaged, you're 
very good at expressing why your idea is good and then it gets launched. 

So, to me a lot of the time when I'm just walking around these confer¬ 
ences and I'm talking to people, I hear good ideas, and I make mental 
notes that I need to engage with them and perhaps get them a little more 
connected into the community. Especially people who are not committers. 
Allan: Yes. 

• Kirk: There's one person who sent me something about a bug report that 
was six years old, and no one had really paid any attention to it, and I just 
went into the source code control and saw who was involved in it. The 
reporter of the bug had included a patch that worked, and he kept updat¬ 
ing it for the new systems. I said, just let it go in. And it did, it went in. 

So, it's little things like that where one email from me makes something 
good happen. I like to take advantage of that. Indeed, that's mostly what 
I'm contributing these days, more than code or anything else. 

Allan: Right. 

Benedict: We touched on the Project itself and the community and also the core 
team that's been elected by the community. But there's also the FreeBSD 
Foundation that you're involved with. Is that also a contributing factor to the 
health of the Project in certain ways? 

• Kirk: Undoubtedly it is. Look at the history of how the FreeBSD Project 
starts. The first distributions were done by Walnut Creek CD-ROM. And 
they paid the initial developers so they could do it as a full-time job, and 
that was a huge step up, but that wasn't a sustaining model because we 
needed a lot more infrastructure and network connectivity. 

And again, sort of fortuitously, we fell into Yahoo. They were using it for 
everything, and they had all the machines and so on. And, they provided 
our infrastructure, and that went on for —/ don't know—more than a 
decade. 

Allan: I think it's still going on. 

• Kirk: It's still going on to some extent. 

Benedict: Sure. 

• Kirk: But the problem is, what happens if Yahoo one day decides to 
switch to Linux—which they're doing. 

Benedict: Or the box, or whatever. 
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• Kirk: Yes, ... whatever. And suddenly our infrastructure disappears from 
underneath us. And so, Justin Gibbs saw this as an issue. He realized that 
the FreeBSD Project needed to have some way of funding at least its basic 
infrastructure. So he started the FreeBSD Foundation, and it took at least 
five or eight years before there was enough income to contemplate even 
basic stuff like infrastructure, and in fact we have only converted over to 
that in the last four or five years. I mean it's been a slow process. But, if 
Yahoo went away today, we'd be fine. 

There are many ways that projects can get infrastructure through 
organizations like GitHub or the Apache Foundation. The FreeBSD Project 
set up our own foundation. By having our own foundation and not being 
under Apache, for example, we have a lot more control over how things 
work, though that can be both good and bad. But overall, the 
Foundation is an important keystone. 

I think the biggest fear of the Project is that somehow the Foundation 
is going to try and take over. To date, the Foundation has done its best to 
position itself so that it is driven by the Project goals rather than trying to 
drive the Project. But the Foundation is important. It's very important. 
Allan: Yes, and at the same time, the Foundation and core are usually the people 
who are able to contribute the most. Both of them are drawing from the same 
feedstock so it's very hard not for them to have some overlap. 

Benedict: Right. Or to have enough trust from the community that they say, 

Okay, that person should be my voice, my representative. 

• Kirk: And so, I like to think that Linux is the dinosaur and FreeBSD is the 
mammal. The meteor is going to hit and then the mammals will take 
over. But I'm not even sure I really want that to happen because if we get 
to be the dominant one, then we're going to fall off. 

Benedict: Yeah, so always a little bit under the radar but swimming in the big 
pond with the other folks. 

Allen: And you know, the rumors of FreeBSD's demise have been exaggerated. 
And part of that is just how FreeBSD gets used and that people don't know 
about it or notice it's there and even to the point where the FreeBSD Project isn't 
always aware of just how much FreeBSD is being used. Or how it's been used. 

• Kirk: Yeah, we get some unexpected contributions to the Foundation 
from companies we had no idea were using FreeBSD. It's like, why did they 
contribute? So of course, the first thing we do is say to the contributor that 
we want to put you on our list of users of FreeBSD. We've had a couple of 
them say. Oh, no, no, no. Absolutely not. We consider the fact that we use 
FreeBSD as a proprietary secret. They don't want to promote that. 

Allan: Yes. 

• Kirk: Or there are others that are using it even though they promote 
themselves as being a Linux shop. And they don't want to NOT be per¬ 
ceived as a Linux shop. But nevertheless, that's what happens. 

Allan: So, where do you see FreeBSD going in the future? 

• Kirk:l/l/e//„ that kind of gets back to this other thing. I think it has a very 
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strong base. I think we can't just take that for granted. Companies fail or 
go into obscurity. Yahoo was our absolute poster child for a long time, 
and now not so much. And that's in part because they are not a complete 
FreeBSD shop anymore, and also because their star is waning in the busi¬ 
ness world. 

So, we need to keep going out there and evangelizing and bringing in 
new companies and new people. But I think we're pretty successful at 
that. I think that we really need to get into more universities, which 
we're trying to do with some success. I've actually been very interested in 
Philip Paeps's efforts. He travels to all these conferences in Africa and Asia 
and other places where you might not normally think of going. There's a 
lot of FreeBSD that's being used out there. Philip is evangelizing FreeBSD 
far and wide. 

Benedict: That's interesting. 

Allan: It reminds me of the mid to late 1990s, almost every ISP was based on 
FreeBSD because it was the easiest and cheapest way to have a reliable dial-up 
server or whatever. And we're seeing some of these developing countries where 
FreeBSD is the answer again. And we just have to do the right things to capital¬ 
ize on that. 

Allan: Because places where Linux hasn't gotten into the market yet, if we can 
just get there first, it means that there's not enough of an advantage to change. 
Benedict: And it's also looking at it from an operating system's perspective, being 
UNIX and that it evolved over these decades and it's also adaptable to different 
systems, different environments to different architectures. 

Allan: Yes, or the fact that the concept of containers was invented on FreeBSD, 
even though it gained its success somewhere else. And some of that is—some¬ 
times we have this bad habit of getting 90% of the way to the right thing and 
then miss that last little bit, to productize it. 

• Kirk: Yes, well, you know what they say, it's the pioneers that get the 
arrows in the back and the settlers that get the land. 

Allan: Yes. 

• Kirk: Unfortunately, we've been pioneers way too much. 

Allan: Yes. 

• Kirk: But I think some things are important if we're going to be success¬ 
ful in the future. We've got to get things like a container solution and 
running in the cloud. The fact that we're on EC2 — first-class on EC2 and 
Azure and other things like that—is definitely important. 

And mostly, tooting our own horn. Android, which everyone talks 
about — oh, you know Android's all Linux. Well, yes, it is a Linux operating 
system but nearly the entire user level on top of that is BSD because the 
GNU license doesn't work for user-level programs. The GPL by design 
forces applications that include its code to release their source code. 

We're in a lot of places we are not well-known for being in. I think we 
need to make a little more of a push to let the people know that. 

Allan: Yes, but I mean you're right that one of the areas we need to push harder 
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on is the universities and so on, so that we get to that next generation of people. 

• Kirk: Right, well, one of the beauties of young people is they like to do 
something different than their parents did. At this point, their parents are 
Linux. 

Benedict: Yes! And it's also giving them the right engineering skills and teaching 
them how to do proper debugging or any kind of skills that are helpful and not 
just in this one area but that they can apply to many others. 

Allen: Well, and that's why I think the Teach BSD Project for university-level things, 
where instead of learning about the concept of computer science in a toy operat¬ 
ing system environment, using a real operating system you can run on your real 
computer to do real work is important. 

• Kirk: Yes, the toy operating system has kind of faded out. But mostly 
what they're learning is Linux. But Linux is now so huge and complex that 
it's really hard to wrap your brain around it, whereas BSD is still at a size 
where you can still understand it. 

Allan: And at the same time, it's that much more observable. We have the right 
tools to actually be able to watch the parts inside while they're moving. 

• Kirk: Yes. 

Allan: Some other machines are big black boxes over in the corner that make a 
lot of noise, but you can't tell what's happening. 

• Kirk: Yes, things like DTrace are much more advanced in FreeBSD than 
they are in, for example, Linux. I saw the DWatch talk today, which is a 
very impressive wrapper around DTrace that brings it to the masses. You 
don't have to even understand most of what DTrace is doing. 

Allan: The hard question has always been coming up with the question to ask, 
not just finding the answer. We have lots of tools to find the answer. But it is you 
that has to decide what the question is. 

• H\rk. Right, but the problem is that DTrace is kind of the assembly lan¬ 
guage for answering questions. All right, I've got this question. But now I 
have to write this 100-line program to answer it. Whereas something like 
DWatch is like, well, I've got this question and 10 lines of stuff and I can 
get a graphic answer. 

Banadict: And that also appeals to more younger folks who want something more 
interesting, like colors and things moving around. 

• Kirk: So, I'm very excited to see stuff like that because I think that's — 
again, it's going to draw in a broader group of people. 

Benedict: Good. 

Allan: Yes. 

Benedict: Well, then, thank you for having this interview with us and hopefully, 
see you again in the next 25 years. Or hopefully sooner than that! 

• Kirk All right, yeah, well, five minutes, huh, no problem. 
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by Benedict Reuschling 

My Year of BSD Conference Travels 

During one of the talks at vBSDcon in September, I had decided to 
skip the current talk to spend some time in the hallway, when Colin 
Percival walked over to me. He started the conversation with: “You've 
been to a lot of events this year.” And, yes, I certainly have been. I 
showed him the directory on my disk where I keep my travel docu¬ 
ments, and it occurred to me that I had been to even more events this 
year than the year before, which had already exceeded the year 
before that. All my trips had something to do with BSD and open 
source, advocating for it, in general, and the OS, in particular. 

From small events to big gatherings, each event had had its own 
charm and reason for me to attend. 

I t started in February with FOSDEM, which is a huge event. A large number of people 
attended the FreeBSD devsummit on the day before. We also had a significant num¬ 
ber of visitors come by the FreeBSD table and to the talks in the BSD devroom at 
FOSDEM. 

* In March, I flew to Tokyo to attend AsiaBSDcon, which is one of the big-three BSD 
conferences (the other two being BSDCan and EuroBSDcon). This was a good opportunity 
for planning new things for the operating systems and showing the work that had already 
been done. It also allowed me to enjoy being a tourist in an amazing country for a few 
days before the conference. 

The Aberdeen hackathon in April was new to me as I've never been to Scotland 
and because it was my first event of this kind. Although it turned out to be small (just the 
three of us, including the organizer, Tom Jones), it was nevertheless productive. I always 
enjoy going to universities other than my own to see how they are organized in terms of 
buildings, classes, equipment, etc. 

A week later, I flew to Graz in Austria to help out at the FreeBSD table at the 
Grazer Linuxtage. This was a good opportunity for me to reconnect with my mentor, 
Johann Kois, who taught me everything I know about how to use the precious doc bit 
effectively. Meeting up with local BSD enthusiasts (no matter how small the group), I was 
reminded that a dedicated group of people can contribute a lot to the Project. Surely, peo¬ 
ple's lives change (family, work, etc.) and they might not have as much time available for 
FreeBSD as they once did, but the enthusiasm is still there. 

May was around the corner by then, and that meant BSDCan. This is an event I look 
forward to every year and for which I make holiday time available as soon as I know the 
dates and can request the time. The amount of energy and enthusiasm across all the vari¬ 
ous BSD projects and people is simply stunning. It's a very intensive week and it's over in 
the blink of an eye (organizers might see this differently), but well worth attending. Allan 
Jude often talks about recharging your BSD batteries at these events. For me, the awe- 
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someness of the whole conference is supercharging! 

^ The next event I attended was another hackathon in Vienna, Austria, in June. Over 
a long weekend, we sat in a room and worked on PRs, committing outstanding patches, 
and reviewing other people's submissions in Phabricator. I also had the opportunity to do 
some one-on-one mentoring, which I always enjoy. It doesn't take long, but you can see 
results almost immediately. Vienna is, of course, a beautiful city and has a lot to offer, 
which made the trip even more worthwhile. 

My only regret this year was that I did not attend the Berlin hackathon at Bally Wulff 
Games. From what I heard, it was a very productive event. Clearly, this is something I have 
to plan for next year. 

* I returned to Vienna in August as a transit stop on my way to COSCUP in Taipei, 
Taiwan. This was a new location for me and the second country I visited in Asia, after 
Japan. It was certainly one of the highlights—experiencing a different culture; the friendli¬ 
ness of the people, how easy it is to navigate around as a foreigner, and the very well 
organized conference all helped me tolerate the 80 % humidity of the season. 

^ In September, I attended the aforementioned vBSDcon in Reston, Virginia, and 
EuroBSDcon, which was held in Lillehammer, Norway, this year. Both were two great con¬ 
ferences, each with its own special style, well organized and well run. They also concluded 
the year of BSD conference travels for me, as I had to teach my Unix class at the begin¬ 
ning of October. If you want to read more details about any of the events just mentioned, 

I encourage you to go back to earlier issues of the FreeBSD Journal to see the full trip 
reports from me or other people. 

2019 turned out to be the year of FreeBSD hackathons. We talked about doing 
them more often last year, and luckily people stepped up to organize them. They provided 
a different venue and a way to get some work done for FreeBSD. My overall impression 
was that people like this sort of event, having the possibility to sit together with someone 
and work on something or casually walk over to another person in the room to ask a 
question. Hackathons are clearly something we should aim to repeat in future years, per¬ 
haps even extend. I saw that it worked well when we did not have enough talks to fill 
two days of devsummit at EuroBSDcon, so we declared the second day a hackathon. 
People showed up and started working on their own little (or sometimes big) projects 
without requiring much outside motivation. 

Now, is the sort of travel I've just described sustainable? Perhaps not, but I'm happy 
that I could do it this year and previously. It's perfectly fine if you cannot attend every¬ 
thing, not even all three of the major BSD conferences. If you can just attend one, that's a 
great start. Organizing even a one-day devsummit or hackathon on your own is not too 
difficult. Even if there are only a handful of people attending the first time, that's okay. 

You might get to know people from your own city that you never knew, but who are just 
as excited about FreeBSD as you are. And if you like this sort of thing, think about doing 
it regularly as sort of a monthly meetup or BSD user group (BUG). 

If you like conferences but don't have the financial resources available, you should 
apply for a travel grant from the FreeBSD Foundation 

( https://www.freebsdfoundation.org/what-we-do/qrants/travel-qrants/ ). We often don't 
get enough applications from people, so don't be shy about filling out the form if you 
really want to go to an event. And don't think, "Oh, I'm too new." One of the goals of 
the grant is that we want to give new people the opportunity to meet BSD folks at these 
gatherings. And it's not only to charge their BSD batteries, but also to give them the 
opportunity to engage with the community. We never know what people do with their 
freshly charged BSD batteries. It could very well be that they'll contribute more to the 
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Project, which certainly makes it a worthwhile investment of the money people have donat¬ 
ed to the Foundation. 

Speaking about the FreeBSD Foundation, I want to make it clear to people who think 
that just because a person currently serves on the Board that their travel is paid for. Nothing 
could be further from the truth. I have never used Foundation funding to attend an event. If 
I can't go somewhere because I can't afford it (or if the conference is not paying for me to 
give a talk), I simply don't go! 

Around the world, I can see that people are excited about the BSDs and that there is still a 
need for more: more advocacy, more support, more people contributing their time in various 
ways, money, and sometimes a good amount of enthusiasm. Even small levels of involve¬ 
ment matter, especially if you are new. 

I received a big package from Li-Wen Hsu at COSCUP as a parting gift for 
attending and giving a talk. I couldn't quite figure out what it was at first, 
and I had to sit on my suitcase just to make it fit. When I got home and un¬ 
packed, I finally saw what it was: he had given me one of the speaker gifts— 
big pillow—from BSD Taiwan that he organized a couple of years ago. 

On one side of it were the words that not only sum up this article, but 
the whole year for me and the reasons why I do these travels: Love BSD. J 
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Benedict tinselling joined the FreeBSD Project in 2009. After receiving his full documentation 
commit bit in 2010, he actively began mentoring other people to become FreeBSD committers. He joined 
the FreeBSD Foundation in 2015, where he is currently serving as vice president. Benedict has a Master of 
Science degree in Computer Science and is teaching a UNIX for software developers class at the Darmstadt 
University of Applied Sciences, Darmstadt, Germany. Together with Allan Jude, he is host of the weekly 
BSDNow.tv (http://BSDNow.tv) podcast. 
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BY DRU LAVIGNE 



This column aims to shine a spotlight on contributors who 
recently received their commit bit and to introduce them to 
the FreeBSD community. In this installment, the spotlight is 
on Santhosh Raju, who received his src bit in August. 



Tell us a bit about yourself, your background, and your 
interests. 

• I am Santhosh Raju (fox@FreeBSD.org) , from India and cur¬ 
rently reside in Ecuador, where I also work at the moment. I 
have been fascinated with computers from quite a young age 
and am fond of working with them. I got introduced to Linux 
when I was in college in 2003. But it was not until 2009 that 
I was introduced to BSD by my friend Cherry G. Mathew. In 
general, I like to play around with both hardware and soft¬ 
ware, and most of that is done during spare time as a hobby. 

I also love to immerse myself in the world of video games 
when the time allows for it. But I have not engaged in any 
active gaming due to lack of a machine to play with. My nickname, fox@), comes 
from an interesting character in one of my favorite video games. 

In the world of computers, I have a general interest in writing and working with 
systems software, though I am yet to have any luck finding a job that fits the 
hobby. Along with Cherry, I got my first taste of working with kernel internals 
when we were writing the portable hotplug API for NetBSD: uvm_hotplug(9) 
[ https://www.netbsd.orq/qallerv/presentations/cherry/eurobsdcon2017/uvm_hotp- 

luq.pdf ], I am also interested in formal verification and have worked a little bit 
with TLA+. 


How did you first learn about FreeBSD and what about FreeBSD interest¬ 
ed you? 

• I was introduced to FreeBSD by Philip Paeps and Tod McQuillin when I met them 
at HillHacks 2015 [ https://hillhacks.in/ 1, a tech conference in the Himalayan 
foothills. Other than trying out NetBSD once in 2009, I really had not dug deep 
into other forms of BSD, even though I had heard of FreeBSD. I was encouraged 
by Philip to try out FreeBSD for hosting an IRC client, as I was running a client 
locally in my laptop and it was annoying to get dropped off whenever I shut down 
my laptop. And since I was eager to try out a Raspberry Pi 2, I thought of running 
FreeBSD/arm 11.0-RELEASE on my RPi2 to set it up as an IRC client using Irssi. It 
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was during this time that I got to explore the details of how to use FreeBSD, 
install packages from either source using the Ports system or the binary package 
manager. Coming from a Linux background, I thought it might be difficult to set 
up and configure things. On the contrary, I found the initial setup and configura¬ 
tion was easier and well organized. In the FreeBSD Handbook, there are suffi¬ 
cient guides and details on various steps to set up the system from basic installa¬ 
tion and initial configuration to setting up ports for third-party software. 

It was during this time I noticed an issue that was preventing me from using 
FreeBSD: my Irssi setup needed a plugin for connecting to ICB networks, which 
was not available in the Ports tree. I ended up compiling the package manually 
from source, but this did not help every time I had to upgrade Irssi. So, with 
some help from Philip and the FreeBSD Handbook I was able to write my first 
contribution to the FreeBSD Ports system, the irc/irssi-icb [ https://bloq.portO.in/ 
20 17/04/my-first-freebsd-portl package. Since then, I have been packaging little 
pieces of software into FreeBSD both as an effort to learn things and to make 
some contributions. Over a period of time, I eventually set up most of the infra¬ 
structure I use in FreeBSD. One of the things I got introduced to by entering into 
BSD land was how to do networking for conferences with computers on a small- 
to-medium scale. Playing around with DHCP, DNS, and pf in a FreeBSD-based 
UniFi controller for the network [ https://bloq.port0.in/2017/07/ 
networkinq-like-the-biq-boys ] was an enjoyable experience and fun exercise. 

One of the things I find useful with FreeBSD is the fact there is ample docu¬ 
mentation tailored both for a first-timer (FreeBSD Handbook) and an experienced 
person who wants to look more into kernel development (man pages). In addi¬ 
tion, upgrading the system and packages is a breeze; building and deploying 
packages for security updates is as easy as bumping version numbers in the Ports 
tree, regenerating the checksum, and installing the update from the package if 
the fix was urgently needed. I have had good experience in building and running 
CURRENT in a virtual machine where I do some development and testing of 
packages. Even when things were half broken: like upgrading an RPi2 over the 
Internet [ https://bloq.port0.in/2018/04/updatinq-freebsd-in-rpi2-remotelv1 — 
where Todd helped me to set up the test RPi2 where I reproduced the problem 
and fixed it—or trying to restore the correct partitions and files for a UniFi con¬ 
troller running on FreeBSD [https://bloq.port0.in/2018/07/a-lesson-in-cloninq 
-disks] , things were not very painful to fix. 


How did you end up becoming a committer? 

• After my first contribution in the form of irc/i rssi-icb, I became interested in 
maintaining packages, whether it was packaging something for a friend or 
something I found useful. Since it was not easy to test changes in my local RPi2 
or my low-spec virtual machine running FreeBSD, Philip helped by giving access 
to his poudriere(8) instance. This is when I truly started to enjoy FreeBSD. 

Building packages in jails and building dependencies from scratch for different 
architectures and different versions of FreeBSD using a package-builder software 
was a very enjoyable experience. This also helped me to test packages I was writ¬ 
ing more efficiently and helped to generate patches for other packages I used 
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that required a fix or a version bump—and all this could be easily tested in 
poudriere. Cliqz [ https://www.freshports.orq/www/cliqz ] is one of the most 
time-consuming ports I help maintain. It is based on Firefox and has some kind 
of interesting issue during every update. However, since it is based on Firefox, I 
can generally keep up with the www/firefox package to see what needs to be 
done to make sure it works correctly. 

After taking on a new job in Ecuador, I tried to keep active by doing a bit of 
maintenance work with the packages during the weekends. Philip suggested I 
should join the ports channel in EFnet and interact with the people there. Even 
though I lurk there a lot, I occasionally interact with people, asking them ques¬ 
tions about Ports. By the end of July, four months after starting the new job, 
due to some very unfortunate circumstances, I ended up being let go from the 
company. With a lot of free time in August 2019, I decided to do bug fixes in 
some packages I used as well as some portlint(l) / portfmt(l) on the packages I 
maintain. It was during this time Philip proposed a commit bit in Ports and vol¬ 
unteered to be my mentor. 


How has your experience been since joining the FreeBSD Project? Do you 
have any advice for readers who may be interested in also becoming a 
FreeBSD committer? 

* It is a bit too early to summarize my experience since joining the FreeBSD 
Project, but since I began using FreeBSD and interacting with the community, it 
has been quite positive. People, in general, are encouraging and helpful and try 
to answer many of the questions I ask. Sometimes I don't get a response, a 
sign probably that people are busy or that I might be asking something too triv¬ 
ial. Much of the time I tend to find the answers in the Handbook, man pages, 
forums, or blogs. 

I think I am a bit too inexperienced to give advice to readers who may be 
interested in becoming a FreeBSD committer, but I could share some things that 
eventually helped me become a committer. 

* The Handbook is my go-to place for reference. Sometimes I do miss things, 
and when I ask questions, I get pointed back to the relevant sections there. 

* Breaking things is fine; one cannot learn until you break something and, 
more importantly, fix it. However, break them under controlled circumstances 
so that the smallest number of people are inconvenienced. 

* When in doubt, ask questions through available channels, mailing lists, ire, or 
slack. It is better to have asked something stupid than to have that doubt 
linger and then do something even more stupid. 

* Taking up responsibilities and making sure that the responsibilities are met 
correctly is an important aspect when a person is handed over a commit bit. 

* A little bit of politeness goes a long way. 


DRU LAVIGNE is a FreeBSD doc committer and the author of 
BSD Hacks and The Best of FreeBSD Basics. 
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WeGet LETTERS 

W W m by Michael W Lucas 



letters@ 

freebscljournal. org 


Dear Wise Person Who Somehow Got Shackled 
Into Answering Letters For the Journal, 

People keep telling me how amazing the ports and packages system 
is, but I keep finding myself having to build my own software instead of 
using packages. It’s a pain. How do they get away with this? Can’t you 
show people that ports and packages aren’t as great as people think? 

Thank you, 

Annoyed Compiler 


Dear Annoyed, 

Everything is terrible. That's the core principle of systems administration. 
Well, maybe not the core principle, but it's certainly sysadmin rule number 
three. And it applies doubly so to packaging systems. 

We wander between packaging systems toting a long list of require¬ 
ments, like automatic dependency installation and upgrades and support 
for our environment's LDAP, YP, KerberoslV and KerberosV authentication 
system, not to mention that some sysadmin back in 1989 declared that 
the organization's official filesystem was a FAT16 release candidate that he 
bootlegged out of the Redmond development lab by clenching the backup 
tape between his mighty buttocks, and you've endured that decision ever 
since because that sysadmin got promoted to Chief Albatross Officer and 
never updated his skills again. And that doesn't even go into the boss's 
fondness for writing monstrously bloated checks to outside vendors while 
whittling employee paychecks anytime she gets bored. 

Or perhaps you have a technological green field. You get to architect 
everything from the ground up and have temporarily deluded yourself into 
believing it will be glorious, flawless, and a joy both to deploy and oper¬ 
ate. Of course, you can't make it quite dead-standard. You want to use 
some special feature everywhere. A feature that will make your environ¬ 
ment perfect for you and intolerable to anyone else, as is your privilege as 
a founder. 

In short: your requirements are unique in precisely the same way that no 
two gangrenous spleens are identical. 

Regardless of your operating system, any packaging system is gleefully 
and maliciously guilty of catering to the Least Common Denominator. 
Package system designers have the same goal as any other person in histo¬ 
ry, which is to make people leave them alone so they can get on with 
what they want to do with the minimum of fuss. This devolves to solving 
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as many problems as possible for as many people as possible. Those of us 
with special requirements—and I'm one of them, you have no idea just how 
special my requirements are—are left banging on the door trying to get in. 
At least fancy nightclubs have bouncers to snootily inform you you're not 
suitable for their packaging system, but they've also shared around my 
photo captioned NOPE, and if that isn't discrimination I don't know what is, 
but still, I much prefer packaging systems that bluntly strong-arm you out of 
the queue and into the conveniently accessible and by some strange chance 
unusually dark alley, behind the dumpster, to use their professionally honed 
skills of targeted violence to explain that you're not welcome. Again. 

Worse, none of this software was ever meant to work together. We've 
got MariaDB and Postgres and Apache and nginx and lighttpd and Rail Road 
Ruby or whatever they call it, where someone had an idea and just had to 
go implement it, never thinking of the innumerate man-hours of collateral 
damage they'd be inflicting on society by trying to make software better. It's 
not that any of these ideas were bad (sure, okay, except systemd and the 
entirety of Oracle, granted), but people keep getting this half-witted hope 
that maybe software doesn't have to be terrible. 

Hope exists to teach young people that there is no hope. 

You, of course, can't help but make it worse. 

You've been handed a screaming mess of an environment built around an 
original 286 running Antediluvian Netware and held together with a glue 
composed of used tea bags and pureed slugs. Every bad decision any of 
your predecessors have ever made haunts you. So you look at this and 
decide what you need is another layer of sun-dried wombat leavings over it 
all. You can call it "rationalizing." That sounds good. And you made your 
decision rationally. Because you're a rational person— 

Ahem. Excuse me. I seem to have coughed up a kidney laughing there. 

Because you've come up with self-justifications for your prejudices and 
called them rationality. That's better. 

So you need packages that support multiple versions of protocols, and a 
filesystem nobody else uses, and probably SNMP, because your bed of nails 
has gotten dull from use and you don't want to escalate to autotrepanation 
for your agony fix. 

You want to take stuff that was never meant to work together, and have 
it behave transparently. 

The shred of good news is, people have done the work before you. 

Other people have decided that MariaDB and Postgres should simultane¬ 
ously integrate with nginx, or Perl, or who knows what. Certain websites 
that shall remain nameless but that rhyme with "Whack Derange" overflow 
with dubious advice toward achieving your nebulously visualized totally-not- 
a-nightmare-l-promise dream. 

And again, I'm just as guilty. I just finished writing a book about the 
Shoggothic Nightmare Misery Protocol, SNMP. My dire research demanded I 
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build net-snmp with features that I was fairly certain no human being had 
ever before attempted. By that sad point I could already see into dimension 
7 ir 2 and I knew the secret name of the so-called "squirrel" who occasionally 
dangles on my office window screen and shrieks his Ode of Eternal Enmity, 
so I went to the ports system determined to carve my unspeakable path. It 
turns out that Eternal Madness was already a build option. (They call it 
"embedded Perl" and "MFD rewrites," but it's eternal madness. You don't 
have to trust me. Those of us who Know Too Much To Ever Be Happy Again 
would welcome your company.) 

No matter what combination of features you need, there's a really good 
chance someone has done it before. 

And that's why the ports and packages system is so valuable. It allows you 
to easily repeat mistakes, both others' and your own. At scale. If you've 
decided that your organization is going to glue universal authentication to 
that Netware server via RADIUS, you can build your own package repository 
that globally enables Radius in everything that supports it. You use the pack¬ 
aging system to distribute your mistakes throughout your little slice of the 
world. And nobody can stop you. 

So, Dear Letter Writer, you are absolutely correct. The ports and packages 
system is terrible. But only because everything is terrible. I would encourage 
you to spend some time learning how it works, only so that you can most 
quickly deploy your innovative new layer over your organization's infrastruc¬ 
ture and earn your successor's undying and well-justified loathing. • 


Have a question for Michael? Send it to letters@fre e bsdiou rn al.com . 


Michael W Lucas (https://mwl.io)'s newest books are Sudo Mastery ; 2nd Edition and 
Terrapin Sky Tango. 



Contact Jim Maurer 
with your article ideas. 

(jmaurer@freebsdjournal.com) 
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BSD Events taking place through March 2020 

BY ANNE DICKISON AND DRU LAVIGNE 


linux.conf.au 2020 • January 13-17, 2020 • Gold Coast, Australia 

k https://linux.conf.au/programme/miniconfs/freebsd/ 

I N I The FreeBSD Foundation, in conjunction with simPRO Software, 


is excited to host a FreeBSD Miniconf on January 14, 2020, at linux.conf.au. 
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FreeBSD Mini Developers Summit • January 31 • Brussels, Belgium 

https://wiki.freebsd.org/DevSummit/2020Q2 

Join fellow FreeBSD developers for a one-day event co-located with FOSDEM 
2020. The summit will consist of discussion sessions and is by-invitation-only. 
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FOSDEM 2020 • February 1 & 2 • Brussels, Belgium https://fosdem.org/2020/ 

0 FOSDEM 2020 offers open-source and free software developers a place to meet, share ideas, 
and collaborate. Renowned for being highly developer-oriented, the event brings together 
some 8,000+ geeks from all over the world. 


APRICOT 2020 • February 12-21 • Melbourne, Australia 

H r - htt ps://2020.apricot.net/ Representing Asia Pacific's largest international Internet confer- 
I— ence, Asia Pacific Regional Internet Conference on Operational Technologies (APRICOT) 
draws many of the world's best Internet engineers, operators, researchers, service providers, 
users, and policy communities from over 50 countries to teach, present, and do their own 
human networking. 
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SCaLE 18x • March 5-8 • Pasadena, CA www.socallinuxexpo.org/scale/18x SCaLE 18x 

The 18th annual Southern California Linux Expo will take place at the Pasadena Convention 
Center. SCaLE 18x expects to host 150 exhibitors, along with nearly 130 sessions, tutorials, 
and special events. The Foundation will again be hosting a one-day FreeBSD workshop. 
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AsiaBSDCon • March 19-22 • Tokyo, Japan https://2020.asiabsdcon.org/ 

AsiaBSDCon is a technical conference for users and developers on BSD-based systems. It is for anyone 
developing, deploying, and using systems based on FreeBSD, NetBSD, OpenBSD, 

DragonFly BSD, Darwin, and MacOS X. AsiaBSDCon aims to collect the best technical 
papers and presentations available to ensure that the latest developments in our 
open-source community are shared with the widest possible audience. 



FOSSASIA Summit 2020 • March 19-22 • Singapore, Singapore 

https://summit.fossasia.org / The FOSSASIA Summit, Asia's leading open-technology 


s 


I 


conference for developers, startups, and IT professionals, will take place at the Lifelong 
Learning Institute Singapore. 
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